From 725c7071c4946f3f52cbc16fffdf61c4c8d79cf8 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Thu, 20 Aug 2026 18:08:04 -0700 Subject: [PATCH 1/9] feat(tests): runnable docker setup for the preview Maestro SDK skills Running the three preview/uipath-maestro-{flow,case,bpmn} builder-SDK skills as the only skill catalog needed four non-obvious pieces of setup, none of them recorded anywhere. Each missing piece scores as a capability failure rather than a config error, so the results look plausible and are not. Dockerfile: add a runtime npmrc for the @uipath scope plus NPM_CONFIG_USERCONFIG. The build-time npmrc is written to globalconfig and deleted in the same RUN because it carries the literal token, so the image ships no npm auth at all; this layer adds a token-LESS npmrc referencing ${NODE_AUTH_TOKEN}, which npm expands at read time. NPM_CONFIG_USERCONFIG is what makes it reachable: npm resolves userconfig to $HOME/.npmrc, and the docker runner forwards --env HOME with the HOST value on purpose, overriding the image's /root, so npm looks in a directory the container does not have, never maps @uipath to GitHub Packages, and 404s on the public registry. Without this, `npm install @uipath/flow-sdk` fails and every builder-SDK compile fails with it. tests/experiments/preview-maestro-sdk.yaml: the catalog is narrowed by pointing agent.plugins[].path at preview/, which shadows the same-named v1 skills. That path does double duty as a bind mount, so narrowing it also drops the repo-root mount that nightly gets for free, and criteria across the flow and case suites that shell out to tests/tasks/**/_shared/*.py then exit 2. extra_mounts restores the repo root explicitly. File-based auth is documented as a commented block: its destination has to equal the host $HOME for the same forwarded-HOME reason, and mounting to /root/.uipath instead yields "Not logged in" on every tenant call. tests/docker/preflight.sh: one container, run with the host HOME forwarded exactly as the harness does, asserting the eleven preconditions a full run depends on -- login state readable, SDK installs, and check/compile/check plus the product scaffold all succeeding. A manual `docker run` without --env HOME authenticates against the image's /root and reproduces none of these failures, which is why they are easy to miss. Co-Authored-By: Claude Opus 5 (1M context) --- tests/docker/Dockerfile | 25 +++++++ tests/docker/preflight.sh | 82 ++++++++++++++++++++++ tests/experiments/preview-maestro-sdk.yaml | 79 +++++++++++++++++++++ 3 files changed, 186 insertions(+) create mode 100755 tests/docker/preflight.sh create mode 100644 tests/experiments/preview-maestro-sdk.yaml diff --git a/tests/docker/Dockerfile b/tests/docker/Dockerfile index 1d1067191b..665add9b1a 100644 --- a/tests/docker/Dockerfile +++ b/tests/docker/Dockerfile @@ -137,3 +137,28 @@ ENV DOTNET_ROOT=/usr/share/dotnet \ # loops in the turn timeline). uipath-langchain is declared in the # SimpleCodedAgent fixture's pyproject.toml but has no pre-built venv. RUN pip install --no-cache-dir "uipath-langchain>=0.9.26" + +# Runtime npm auth for the @uipath scope, needed by any task whose skill tells +# the agent to `npm install @uipath/...` -- the Maestro builder SDK +# (`@uipath/flow-sdk`) is the live case, and it is published ONLY to GitHub +# Packages. +# +# The build-time npmrc above is written to `npm config get globalconfig` and +# deleted in the same RUN, on purpose: it carries the literal token. This layer +# writes a token-LESS npmrc instead, referencing ${NODE_AUTH_TOKEN}, which npm +# expands at read time. Nothing secret enters the image; the eval harness +# forwards the variable (env_passthrough_extra: NODE_AUTH_TOKEN). +# +# NPM_CONFIG_USERCONFIG is the half that actually decides whether this works, +# and the reason is not obvious. npm resolves `userconfig` to $HOME/.npmrc, and the docker runner +# forwards --env HOME with the HOST value by design (coder_eval +# models/sandbox.py: "HOME is intentional in default"), which overrides this +# image's /root. So npm looks for /home//.npmrc, finds nothing, never +# maps the @uipath scope to GitHub Packages, and falls through to the public +# registry -- where the package 404s. Pinning userconfig makes npm's view of +# the npmrc independent of whatever HOME the harness forwards. +RUN printf '%s\n' \ + '@uipath:registry=https://npm.pkg.github.com/' \ + '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' \ + > /root/.npmrc +ENV NPM_CONFIG_USERCONFIG=/root/.npmrc diff --git a/tests/docker/preflight.sh b/tests/docker/preflight.sh new file mode 100755 index 0000000000..e67f9bd34b --- /dev/null +++ b/tests/docker/preflight.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Pre-flight gate for a docker-driver run of the Maestro builder-SDK skills. +# +# Runs ONE container that reproduces the harness environment and asserts the +# preconditions a 250-row run depends on. Every check here stands for a failure +# that has actually happened and that scores as a capability problem rather than +# a config one, so it is invisible in the results: +# +# * npm cannot reach GitHub Packages -> "@uipath/flow-sdk is not installed" +# on every compile +# * the login state is not readable -> "Not logged in" on every tenant call +# * the skills repo root is unmounted -> criteria that shell out to +# tests/tasks/**/_shared/*.py exit 2 +# +# The single most important detail: pass --env HOME with the HOST value, exactly +# as the runner does. A manual `docker run` WITHOUT it authenticates fine +# against the image's /root and reproduces nothing. +# +# Usage: tests/docker/preflight.sh [image] [uipath-home] +# image default skills-codex:latest +# uipath-home host dir holding .uipath login state, default $HOME/.uipath +# +# Requires NODE_AUTH_TOKEN (GitHub Packages read:packages, SSO-authorized). +set -uo pipefail + +IMG="${1:-skills-codex:latest}" +UIPATH_HOME="${2:-$HOME/.uipath}" +SKILLS_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +: "${NODE_AUTH_TOKEN:?NODE_AUTH_TOKEN must be set (GitHub Packages token)}" +[ -d "$UIPATH_HOME" ] || { echo "FAIL: no uipath home at $UIPATH_HOME"; exit 1; } + +LOG="$(mktemp)" +trap 'rm -f "$LOG"' EXIT + +docker run --rm \ + --env HOME="$HOME" \ + --env NODE_AUTH_TOKEN \ + --env UIPATH_CLI_DISABLE_VERSION_SYNC=1 \ + -v "$UIPATH_HOME:$HOME/.uipath:rw" \ + -v "$SKILLS_REPO:$SKILLS_REPO:ro" \ + --entrypoint bash "$IMG" -c ' + echo "HOME=$HOME"; node -v; echo "uip $(uip --version 2>&1 | tail -1)" + echo "userconfig=$(npm config get userconfig)" + echo "uipath_registry=$(npm config get @uipath:registry)" + echo "token_seen=$([ -n "${NODE_AUTH_TOKEN:-}" ] && echo yes || echo no)" + uip login status 2>&1 | head -12 + d=$(mktemp -d); cd "$d"; npm init -y >/dev/null 2>&1 + npm install @uipath/flow-sdk 2>&1 | tail -2 + echo "SDK_VERSION=$(node -e "console.log(require(\"@uipath/flow-sdk/package.json\").version)" 2>&1)" + cat > Hello.flow.ts </dev/null 2>&1; echo "RC_CHECK_SOURCE=$?" + uip maestro flow compile Hello -o Hello.flow >/dev/null 2>&1; echo "RC_COMPILE=$?" + [ -s Hello.flow ] && echo "EMITTED=yes" || echo "EMITTED=no" + uip maestro flow check Hello.flow --compiled >/dev/null 2>&1; echo "RC_CHECK_COMPILED=$?" + uip solution init HelloSol >/dev/null 2>&1 + ( cd HelloSol && uip maestro flow init Hello >/dev/null 2>&1 ) + [ -f HelloSol/Hello/project.uiproj ] && echo "SCAFFOLD=yes" || echo "SCAFFOLD=no" + ' >"$LOG" 2>&1 + +fail=0 +chk() { if grep -qE "$2" "$LOG"; then echo " PASS $1"; else echo " FAIL $1"; fail=1; fi; } +echo "=== pre-flight: $IMG ===" +chk "container HOME is the forwarded host HOME" "^HOME=$HOME\$" +chk "npm userconfig is HOME-independent" '^userconfig=/root/\.npmrc$' +chk "@uipath scope resolves to GitHub Packages" '^uipath_registry=https://npm\.pkg\.github\.com/?$' +chk "GH Packages token reached the container" '^token_seen=yes$' +chk "uip reports a live login" '"Status": "Logged in"' +chk "flow-sdk installs in-sandbox" '^SDK_VERSION=[0-9]+\.[0-9]+\.[0-9]+$' +chk "flow check --source exits 0" '^RC_CHECK_SOURCE=0$' +chk "flow compile exits 0" '^RC_COMPILE=0$' +chk "compile emitted an artifact" '^EMITTED=yes$' +chk "flow check --compiled exits 0" '^RC_CHECK_COMPILED=0$' +chk "product scaffold emits project.uiproj" '^SCAFFOLD=yes$' +echo " $(grep -m1 '^SDK_VERSION=' "$LOG") $(grep -m1 '^uip ' "$LOG")" +if [ "$fail" -ne 0 ]; then echo "GATE=FAIL"; echo "--- container output ---"; cat "$LOG"; exit 1; fi +echo "GATE=PASS" diff --git a/tests/experiments/preview-maestro-sdk.yaml b/tests/experiments/preview-maestro-sdk.yaml new file mode 100644 index 0000000000..c6cd963497 --- /dev/null +++ b/tests/experiments/preview-maestro-sdk.yaml @@ -0,0 +1,79 @@ +experiment_id: preview-maestro-sdk +description: >- + The three preview Maestro builder-SDK skills (uipath-maestro-{flow,case,bpmn}) + as the ONLY skill catalog, under the docker driver. Shadows the shipped v1 + skills of the same name by pointing the catalog at preview/ alone, so a run + measures the SDK authoring path rather than a mix of both generations. + Pair with `tests/docker/preflight.sh`, which asserts the preconditions this + file depends on before a full run is launched. + +defaults: + run_limits: + max_turns: 200 + task_timeout: 1200 + turn_timeout: 900 + + sandbox: + driver: docker + docker: + # Build with tests/docker/Dockerfile. The npmrc + NPM_CONFIG_USERCONFIG + # layer there is a hard requirement: without it, in-sandbox + # `npm install @uipath/flow-sdk` 404s and every compile fails. + image: skills-codex:latest + network: bridge + env_passthrough_extra: + - SKILLS_REPO_PATH + # GitHub Packages auth for `npm install @uipath/flow-sdk`. The image's + # npmrc references ${NODE_AUTH_TOKEN} and expands it at read time, so the + # token lives only in the environment. + - NODE_AUTH_TOKEN + - UIPATH_CLI_DISABLE_VERSION_SYNC + - UIPATH_CLI_ENABLE_ENV_AUTH + - UIPATH_CLI_AUTH_TOKEN + - UIPATH_CLI_ORGANIZATION_NAME + - UIPATH_CLI_ORGANIZATION_ID + - UIPATH_CLI_TENANT_NAME + - UIPATH_CLI_TENANT_ID + - E2E_PROCESS_KEY + - E2E_LONG_PROCESS_KEY + extra_mounts: + # The skills REPO ROOT, read-only at its host path. Nightly gets this for + # free because its plugins.path IS the repo root. Narrowing the catalog to + # preview/ drops it, and criteria across the flow and case suites shell out + # to $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py, which then exit 2 with + # "No such file or directory" and score as ordinary misses. Mounting is not + # loading: skills still come only from agent.plugins[].path below. + - $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro + # + # FILE-BASED AUTH, uncomment on a host that has no UIPATH_CLI_ENABLE_ENV_AUTH + # (a dev box or the eval VM, as opposed to CI which passes a token in the + # environment). The destination must equal the HOST $HOME, because the runner + # forwards --env HOME with the host value, which overrides the image's /root, + # and `uip` then looks for $HOME/.uipath. Mounting to /root/.uipath instead + # silently yields "Not logged in" on every tenant call. + # + # - ~/.uipath:$HOME/.uipath:rw + # + # Destination expansion needs coder_eval with `$VAR`/`~` support on the + # mount destination. On an older version, write your host home literally: + # + # - ~/.uipath:/home/youruser/.uipath:rw + + agent: + permission_mode: acceptEdits + allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep"] + plugins: + # PREVIEW ONLY. This path is BOTH the skill catalog and an automatic :ro + # bind mount at the same absolute path in-container, with $VAR expanded. + - type: "local" + path: "$SKILLS_REPO_PATH/preview" + ignore_patterns: [] + + post_run: + # A per-task `npm install` of the SDK leaves node_modules behind in every + # artifact dir; unpruned, a full Maestro run costs gigabytes of run output. + - command: "find . -maxdepth 5 -type d \\( -name node_modules -o -name .npm-prefix -o -name .venv \\) -prune -exec rm -rf {} +" + timeout: 30 + +variants: + - variant_id: default From 28ef73779e94aaffc4983d748249a170b2e59fdf Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 17:50:33 -0700 Subject: [PATCH 2/9] refactor(tests): unify auth on the nightly mount, rename to flow-v2-* The preview experiment shipped its login mount commented out with a `$HOME/.uipath` destination, blocked on destination-side `$VAR` expansion (UiPath/coder_eval#128). It doesn't need it: nightly.yaml and smoke.yaml already authenticate the in-container `uip` from `~/.uipath:/.uipath:rw`, a literal destination that validates on the pinned coder_eval 0.10.2. Adopt that mount verbatim rather than deriving a "more correct" one. It is the arrangement with a 500-task/night track record, and a login the CLI cannot see fails tasks on their tenant calls, which scores as a capability problem rather than a config error. flow-v2-preflight.sh mounts the same destination, so its "uip reports a live login" check is now the empirical test of the unified path. same-ground-headtohead.yaml gets the same treatment, dropping both `/home/tmatup/...` hardcodes and the comment deferring them to UiPath/coder_eval#100. One `$VAR` destination survives, on the repo-root mount, and it is unrelated to auth: DockerDriverConfig can forward host env vars but cannot set container ones, so criteria see `$SKILLS_REPO_PATH` with its forwarded host value and the mount has to land there. That single line is what still wants coder_eval#128. Renames make the Flow v2 scope legible: tests/experiments/preview-maestro-sdk.yaml -> flow-v2-preview.yaml tests/docker/preflight.sh -> flow-v2-preflight.sh same-ground-headtohead.yaml keeps its name: per tests/README.md it is a general campaign comparison arm (pinned skills-image:sg1, SG_EMPTY_SKILLS blanking installed skills), not the v1 half of a Flow v2 head-to-head. Co-Authored-By: Claude Opus 5 (1M context) --- tests/README.md | 13 ++++++ .../{preflight.sh => flow-v2-preflight.sh} | 13 ++++-- ...-maestro-sdk.yaml => flow-v2-preview.yaml} | 45 ++++++++++--------- tests/experiments/same-ground-headtohead.yaml | 10 +++-- 4 files changed, 53 insertions(+), 28 deletions(-) rename tests/docker/{preflight.sh => flow-v2-preflight.sh} (87%) rename tests/experiments/{preview-maestro-sdk.yaml => flow-v2-preview.yaml} (59%) diff --git a/tests/README.md b/tests/README.md index 9148e31780..ac274949fa 100644 --- a/tests/README.md +++ b/tests/README.md @@ -179,6 +179,7 @@ Run-time caps live under `defaults.run_limits` (see coder_eval `RunLimits`). | `smoke-windows.yaml` | tempdir | PR-gate smoke (Windows RPA only) | 40 | 900s | 900s | | `activation.yaml` | tempdir | Skill activation classifier (benchmark) | 3 + early-stop | 360s | 120s | | `same-ground-headtohead.yaml` | docker | Campaign-only local comparison arm | 200 | 1200s | 900s | +| `flow-v2-preview.yaml` | docker | Flow v2 builder-SDK preview skills | 200 | 1200s | 900s | `same-ground-headtohead.yaml` is not a clean-checkout CI experiment. The campaign runner first builds the pinned `skills-image:sg1`, prepares isolated @@ -189,6 +190,18 @@ runner. The image build passes the package credential as exists only for the external nightly caller during migration. Regular nightly and smoke jobs continue to use `skills-image:latest`. +`flow-v2-preview.yaml` runs the three `preview/uipath-maestro-{flow,case,bpmn}` +builder-SDK skills as the ONLY skill catalog, shadowing the shipped v1 skills of +the same name, so a run measures the Flow v2 authoring path rather than a mix of +both generations. Two consequences of narrowing `plugins.path` to `preview/`: +the automatic repo-root bind mount goes with it, so the repo root is remounted +explicitly (criteria shell out to `$SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py`), +and the image needs runtime npm auth for the `@uipath` scope because tasks +`npm install @uipath/flow-sdk` in-sandbox. Login state mounts at `/.uipath`, +identical to `nightly.yaml`. Run `tests/docker/flow-v2-preflight.sh` first: it +asserts those preconditions in one container, each standing for a failure that +otherwise scores as a capability problem rather than a config one. + `activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.9.1. It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md). For **A/B comparisons between two skill variants** (e.g. `main` vs a feature branch, or two historical commits), see [`experiments/skill-comparison-playbook.md`](experiments/skill-comparison-playbook.md) and the [`experiments/skill-comparison-template.yaml`](experiments/skill-comparison-template.yaml). The playbook covers worktree setup, SHA pinning for reproducibility, getting N>1, and interpreting divergent tasks. To automate the whole flow, use the `/skill-compare [task_selector] [n_reps]` slash command — each ref can be a branch name or a commit SHA, and `task_selector` accepts a skill name (`uipath-maestro-flow`), tag list (`tags:smoke,init`), or path globs (`paths:tasks/uipath-maestro-flow/*.yaml`). diff --git a/tests/docker/preflight.sh b/tests/docker/flow-v2-preflight.sh similarity index 87% rename from tests/docker/preflight.sh rename to tests/docker/flow-v2-preflight.sh index e67f9bd34b..68202931f5 100755 --- a/tests/docker/preflight.sh +++ b/tests/docker/flow-v2-preflight.sh @@ -1,5 +1,6 @@ #!/usr/bin/env bash -# Pre-flight gate for a docker-driver run of the Maestro builder-SDK skills. +# Pre-flight gate for a docker-driver run of the Flow v2 (Maestro builder-SDK) +# preview skills. Pairs with tests/experiments/flow-v2-preview.yaml. # # Runs ONE container that reproduces the harness environment and asserts the # preconditions a 250-row run depends on. Every check here stands for a failure @@ -16,7 +17,13 @@ # as the runner does. A manual `docker run` WITHOUT it authenticates fine # against the image's /root and reproduces nothing. # -# Usage: tests/docker/preflight.sh [image] [uipath-home] +# The login mount is `:/.uipath:rw`, byte-for-byte what nightly.yaml uses. That +# destination is the empirical one, not a derived one: it is the arrangement with +# a 500-task/night track record, and the "uip reports a live login" check below +# is what confirms it still holds for this image. Do not "fix" it to +# $HOME/.uipath without re-running this gate. +# +# Usage: tests/docker/flow-v2-preflight.sh [image] [uipath-home] # image default skills-codex:latest # uipath-home host dir holding .uipath login state, default $HOME/.uipath # @@ -36,7 +43,7 @@ docker run --rm \ --env HOME="$HOME" \ --env NODE_AUTH_TOKEN \ --env UIPATH_CLI_DISABLE_VERSION_SYNC=1 \ - -v "$UIPATH_HOME:$HOME/.uipath:rw" \ + -v "$UIPATH_HOME:/.uipath:rw" \ -v "$SKILLS_REPO:$SKILLS_REPO:ro" \ --entrypoint bash "$IMG" -c ' echo "HOME=$HOME"; node -v; echo "uip $(uip --version 2>&1 | tail -1)" diff --git a/tests/experiments/preview-maestro-sdk.yaml b/tests/experiments/flow-v2-preview.yaml similarity index 59% rename from tests/experiments/preview-maestro-sdk.yaml rename to tests/experiments/flow-v2-preview.yaml index c6cd963497..bc2839fbcb 100644 --- a/tests/experiments/preview-maestro-sdk.yaml +++ b/tests/experiments/flow-v2-preview.yaml @@ -1,11 +1,11 @@ -experiment_id: preview-maestro-sdk +experiment_id: flow-v2-preview description: >- - The three preview Maestro builder-SDK skills (uipath-maestro-{flow,case,bpmn}) - as the ONLY skill catalog, under the docker driver. Shadows the shipped v1 - skills of the same name by pointing the catalog at preview/ alone, so a run - measures the SDK authoring path rather than a mix of both generations. - Pair with `tests/docker/preflight.sh`, which asserts the preconditions this - file depends on before a full run is launched. + Flow v2: the three preview Maestro builder-SDK skills + (uipath-maestro-{flow,case,bpmn}) as the ONLY skill catalog, under the docker + driver. Shadows the shipped v1 skills of the same name by pointing the catalog + at preview/ alone, so a run measures the SDK authoring path rather than a mix + of both generations. Pair with `tests/docker/flow-v2-preflight.sh`, which + asserts the preconditions this file depends on before a full run is launched. defaults: run_limits: @@ -28,6 +28,9 @@ defaults: # token lives only in the environment. - NODE_AUTH_TOKEN - UIPATH_CLI_DISABLE_VERSION_SYNC + # Same set nightly.yaml forwards. Nothing in the eval infra sets these; + # they are the runtime env-auth path UiPath Delegate supplies, and are + # inert (silently omitted) when unset. Kept for parity with nightly. - UIPATH_CLI_ENABLE_ENV_AUTH - UIPATH_CLI_AUTH_TOKEN - UIPATH_CLI_ORGANIZATION_NAME @@ -37,27 +40,27 @@ defaults: - E2E_PROCESS_KEY - E2E_LONG_PROCESS_KEY extra_mounts: + # uip CLI login state. Byte-for-byte the mount nightly.yaml uses, and + # deliberately so: this is the one auth arrangement with a 500-task/night + # track record, and a login the CLI cannot see fails tasks on their tenant + # calls, which scores as a capability problem rather than a config one. + # Verify with tests/docker/flow-v2-preflight.sh ("uip reports a live + # login") rather than reasoning about it. + - ~/.uipath:/.uipath:rw + # # The skills REPO ROOT, read-only at its host path. Nightly gets this for # free because its plugins.path IS the repo root. Narrowing the catalog to # preview/ drops it, and criteria across the flow and case suites shell out # to $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py, which then exit 2 with # "No such file or directory" and score as ordinary misses. Mounting is not # loading: skills still come only from agent.plugins[].path below. - - $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro - # - # FILE-BASED AUTH, uncomment on a host that has no UIPATH_CLI_ENABLE_ENV_AUTH - # (a dev box or the eval VM, as opposed to CI which passes a token in the - # environment). The destination must equal the HOST $HOME, because the runner - # forwards --env HOME with the host value, which overrides the image's /root, - # and `uip` then looks for $HOME/.uipath. Mounting to /root/.uipath instead - # silently yields "Not logged in" on every tenant call. # - # - ~/.uipath:$HOME/.uipath:rw - # - # Destination expansion needs coder_eval with `$VAR`/`~` support on the - # mount destination. On an older version, write your host home literally: - # - # - ~/.uipath:/home/youruser/.uipath:rw + # The destination must be the HOST path, because $SKILLS_REPO_PATH is + # forwarded with its host value and the criteria interpolate it in-container. + # Writing that portably needs destination-side $VAR expansion, i.e. + # coder_eval > 0.11.1 (UiPath/coder_eval#128). This is the only line in this + # file that needs it. + - $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro agent: permission_mode: acceptEdits diff --git a/tests/experiments/same-ground-headtohead.yaml b/tests/experiments/same-ground-headtohead.yaml index 1cab1996b7..21061c04bf 100644 --- a/tests/experiments/same-ground-headtohead.yaml +++ b/tests/experiments/same-ground-headtohead.yaml @@ -28,10 +28,12 @@ defaults: - SKILLS_REPO_PATH - UIPATH_CLI_DISABLE_VERSION_SYNC extra_mounts: - # coder-eval forwards the host HOME into this image, so the destination - # must match that path until UiPath/coder_eval#100 makes HOME portable. - - ${SG_UIPATH_HOME}:/home/tmatup/.uipath:rw - - ${SG_EMPTY_SKILLS}:/home/tmatup/.uipath/.skills:ro + # Same login destination nightly.yaml uses. This used to be a literal + # /home/tmatup/... because coder-eval forwards the host HOME and there is + # no destination-side $VAR expansion; the nightly path needs neither, so + # the hardcode is gone. Deeper target wins, so .skills still shadows. + - ${SG_UIPATH_HOME}:/.uipath:rw + - ${SG_EMPTY_SKILLS}:/.uipath/.skills:ro agent: type: claude-code From 968ef96cdc1a80f99ed7a4b7581251b178e66c91 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 17:59:00 -0700 Subject: [PATCH 3/9] docs(tests): trim flow-v2 comments and drop product checks from preflight Comment volume was out of proportion to the config it explains, in the experiment, same-ground, the Dockerfile npmrc layer and the README. flow-v2-preflight.sh loses its six product-behaviour checks (flow check --source / compile / emitted / check --compiled / solution init / scaffold) and the inline TS flow they needed. Those are what the eval measures, and pinning CLI verbs in a gate script only rots. What remains is the six config assertions whose failures score as capability problems instead: forwarded HOME, npm userconfig, @uipath registry, token reachability, a live uip login, and an in-sandbox flow-sdk install. 82 lines to 52. Co-Authored-By: Claude Opus 5 (1M context) --- tests/README.md | 8 +-- tests/docker/Dockerfile | 26 +++------ tests/docker/flow-v2-preflight.sh | 55 +++---------------- tests/experiments/flow-v2-preview.yaml | 42 ++++---------- tests/experiments/same-ground-headtohead.yaml | 6 +- 5 files changed, 32 insertions(+), 105 deletions(-) diff --git a/tests/README.md b/tests/README.md index ac274949fa..2eae26c7c5 100644 --- a/tests/README.md +++ b/tests/README.md @@ -193,11 +193,9 @@ and smoke jobs continue to use `skills-image:latest`. `flow-v2-preview.yaml` runs the three `preview/uipath-maestro-{flow,case,bpmn}` builder-SDK skills as the ONLY skill catalog, shadowing the shipped v1 skills of the same name, so a run measures the Flow v2 authoring path rather than a mix of -both generations. Two consequences of narrowing `plugins.path` to `preview/`: -the automatic repo-root bind mount goes with it, so the repo root is remounted -explicitly (criteria shell out to `$SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py`), -and the image needs runtime npm auth for the `@uipath` scope because tasks -`npm install @uipath/flow-sdk` in-sandbox. Login state mounts at `/.uipath`, +both generations. Narrowing `plugins.path` to `preview/` drops the automatic +repo-root bind mount, so the root is remounted explicitly; the image also needs +runtime npm auth for the `@uipath` scope. Login state mounts at `/.uipath`, identical to `nightly.yaml`. Run `tests/docker/flow-v2-preflight.sh` first: it asserts those preconditions in one container, each standing for a failure that otherwise scores as a capability problem rather than a config one. diff --git a/tests/docker/Dockerfile b/tests/docker/Dockerfile index 665add9b1a..6d14547303 100644 --- a/tests/docker/Dockerfile +++ b/tests/docker/Dockerfile @@ -138,25 +138,15 @@ ENV DOTNET_ROOT=/usr/share/dotnet \ # SimpleCodedAgent fixture's pyproject.toml but has no pre-built venv. RUN pip install --no-cache-dir "uipath-langchain>=0.9.26" -# Runtime npm auth for the @uipath scope, needed by any task whose skill tells -# the agent to `npm install @uipath/...` -- the Maestro builder SDK -# (`@uipath/flow-sdk`) is the live case, and it is published ONLY to GitHub -# Packages. +# Runtime npm auth for the @uipath scope, needed by any task that runs +# `npm install @uipath/...` in-sandbox (the Maestro builder SDK, +# `@uipath/flow-sdk`, is published only to GitHub Packages). # -# The build-time npmrc above is written to `npm config get globalconfig` and -# deleted in the same RUN, on purpose: it carries the literal token. This layer -# writes a token-LESS npmrc instead, referencing ${NODE_AUTH_TOKEN}, which npm -# expands at read time. Nothing secret enters the image; the eval harness -# forwards the variable (env_passthrough_extra: NODE_AUTH_TOKEN). -# -# NPM_CONFIG_USERCONFIG is the half that actually decides whether this works, -# and the reason is not obvious. npm resolves `userconfig` to $HOME/.npmrc, and the docker runner -# forwards --env HOME with the HOST value by design (coder_eval -# models/sandbox.py: "HOME is intentional in default"), which overrides this -# image's /root. So npm looks for /home//.npmrc, finds nothing, never -# maps the @uipath scope to GitHub Packages, and falls through to the public -# registry -- where the package 404s. Pinning userconfig makes npm's view of -# the npmrc independent of whatever HOME the harness forwards. +# Token-LESS: npm expands ${NODE_AUTH_TOKEN} at read time, so nothing secret +# enters the image. NPM_CONFIG_USERCONFIG is required, not belt-and-braces: +# npm resolves userconfig to $HOME/.npmrc, and the runner forwards --env HOME +# with the HOST value, so npm would otherwise look in a dir the container does +# not have and fall through to the public registry, where the package 404s. RUN printf '%s\n' \ '@uipath:registry=https://npm.pkg.github.com/' \ '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' \ diff --git a/tests/docker/flow-v2-preflight.sh b/tests/docker/flow-v2-preflight.sh index 68202931f5..03f3aa7986 100755 --- a/tests/docker/flow-v2-preflight.sh +++ b/tests/docker/flow-v2-preflight.sh @@ -1,32 +1,15 @@ #!/usr/bin/env bash -# Pre-flight gate for a docker-driver run of the Flow v2 (Maestro builder-SDK) -# preview skills. Pairs with tests/experiments/flow-v2-preview.yaml. +# Config gate for a docker run of tests/experiments/flow-v2-preview.yaml. # -# Runs ONE container that reproduces the harness environment and asserts the -# preconditions a 250-row run depends on. Every check here stands for a failure -# that has actually happened and that scores as a capability problem rather than -# a config one, so it is invisible in the results: +# Asserts only the things that, when wrong, score as capability failures +# instead of config errors. Product behaviour (flow check/compile/scaffold) is +# deliberately NOT checked here: the eval measures that, and asserting CLI +# verbs here just rots. # -# * npm cannot reach GitHub Packages -> "@uipath/flow-sdk is not installed" -# on every compile -# * the login state is not readable -> "Not logged in" on every tenant call -# * the skills repo root is unmounted -> criteria that shell out to -# tests/tasks/**/_shared/*.py exit 2 -# -# The single most important detail: pass --env HOME with the HOST value, exactly -# as the runner does. A manual `docker run` WITHOUT it authenticates fine -# against the image's /root and reproduces nothing. -# -# The login mount is `:/.uipath:rw`, byte-for-byte what nightly.yaml uses. That -# destination is the empirical one, not a derived one: it is the arrangement with -# a 500-task/night track record, and the "uip reports a live login" check below -# is what confirms it still holds for this image. Do not "fix" it to -# $HOME/.uipath without re-running this gate. +# Pass --env HOME with the HOST value, as the runner does. Without it a manual +# `docker run` authenticates against the image's /root and reproduces nothing. # # Usage: tests/docker/flow-v2-preflight.sh [image] [uipath-home] -# image default skills-codex:latest -# uipath-home host dir holding .uipath login state, default $HOME/.uipath -# # Requires NODE_AUTH_TOKEN (GitHub Packages read:packages, SSO-authorized). set -uo pipefail @@ -46,28 +29,14 @@ docker run --rm \ -v "$UIPATH_HOME:/.uipath:rw" \ -v "$SKILLS_REPO:$SKILLS_REPO:ro" \ --entrypoint bash "$IMG" -c ' - echo "HOME=$HOME"; node -v; echo "uip $(uip --version 2>&1 | tail -1)" + echo "HOME=$HOME" echo "userconfig=$(npm config get userconfig)" echo "uipath_registry=$(npm config get @uipath:registry)" echo "token_seen=$([ -n "${NODE_AUTH_TOKEN:-}" ] && echo yes || echo no)" uip login status 2>&1 | head -12 d=$(mktemp -d); cd "$d"; npm init -y >/dev/null 2>&1 - npm install @uipath/flow-sdk 2>&1 | tail -2 + npm install @uipath/flow-sdk >/dev/null 2>&1 echo "SDK_VERSION=$(node -e "console.log(require(\"@uipath/flow-sdk/package.json\").version)" 2>&1)" - cat > Hello.flow.ts </dev/null 2>&1; echo "RC_CHECK_SOURCE=$?" - uip maestro flow compile Hello -o Hello.flow >/dev/null 2>&1; echo "RC_COMPILE=$?" - [ -s Hello.flow ] && echo "EMITTED=yes" || echo "EMITTED=no" - uip maestro flow check Hello.flow --compiled >/dev/null 2>&1; echo "RC_CHECK_COMPILED=$?" - uip solution init HelloSol >/dev/null 2>&1 - ( cd HelloSol && uip maestro flow init Hello >/dev/null 2>&1 ) - [ -f HelloSol/Hello/project.uiproj ] && echo "SCAFFOLD=yes" || echo "SCAFFOLD=no" ' >"$LOG" 2>&1 fail=0 @@ -79,11 +48,5 @@ chk "@uipath scope resolves to GitHub Packages" '^uipath_registry=https://npm\.p chk "GH Packages token reached the container" '^token_seen=yes$' chk "uip reports a live login" '"Status": "Logged in"' chk "flow-sdk installs in-sandbox" '^SDK_VERSION=[0-9]+\.[0-9]+\.[0-9]+$' -chk "flow check --source exits 0" '^RC_CHECK_SOURCE=0$' -chk "flow compile exits 0" '^RC_COMPILE=0$' -chk "compile emitted an artifact" '^EMITTED=yes$' -chk "flow check --compiled exits 0" '^RC_CHECK_COMPILED=0$' -chk "product scaffold emits project.uiproj" '^SCAFFOLD=yes$' -echo " $(grep -m1 '^SDK_VERSION=' "$LOG") $(grep -m1 '^uip ' "$LOG")" if [ "$fail" -ne 0 ]; then echo "GATE=FAIL"; echo "--- container output ---"; cat "$LOG"; exit 1; fi echo "GATE=PASS" diff --git a/tests/experiments/flow-v2-preview.yaml b/tests/experiments/flow-v2-preview.yaml index bc2839fbcb..1e8097008b 100644 --- a/tests/experiments/flow-v2-preview.yaml +++ b/tests/experiments/flow-v2-preview.yaml @@ -16,21 +16,15 @@ defaults: sandbox: driver: docker docker: - # Build with tests/docker/Dockerfile. The npmrc + NPM_CONFIG_USERCONFIG - # layer there is a hard requirement: without it, in-sandbox - # `npm install @uipath/flow-sdk` 404s and every compile fails. + # Build with tests/docker/Dockerfile: its npmrc layer is required. image: skills-codex:latest network: bridge env_passthrough_extra: - SKILLS_REPO_PATH - # GitHub Packages auth for `npm install @uipath/flow-sdk`. The image's - # npmrc references ${NODE_AUTH_TOKEN} and expands it at read time, so the - # token lives only in the environment. + # GitHub Packages auth for `npm install @uipath/flow-sdk`. - NODE_AUTH_TOKEN - UIPATH_CLI_DISABLE_VERSION_SYNC - # Same set nightly.yaml forwards. Nothing in the eval infra sets these; - # they are the runtime env-auth path UiPath Delegate supplies, and are - # inert (silently omitted) when unset. Kept for parity with nightly. + # Parity with nightly.yaml. Inert unless a Delegate runtime sets them. - UIPATH_CLI_ENABLE_ENV_AUTH - UIPATH_CLI_AUTH_TOKEN - UIPATH_CLI_ORGANIZATION_NAME @@ -40,41 +34,25 @@ defaults: - E2E_PROCESS_KEY - E2E_LONG_PROCESS_KEY extra_mounts: - # uip CLI login state. Byte-for-byte the mount nightly.yaml uses, and - # deliberately so: this is the one auth arrangement with a 500-task/night - # track record, and a login the CLI cannot see fails tasks on their tenant - # calls, which scores as a capability problem rather than a config one. - # Verify with tests/docker/flow-v2-preflight.sh ("uip reports a live - # login") rather than reasoning about it. + # uip login state. Same destination nightly.yaml uses; do not "fix" it. - ~/.uipath:/.uipath:rw - # - # The skills REPO ROOT, read-only at its host path. Nightly gets this for - # free because its plugins.path IS the repo root. Narrowing the catalog to - # preview/ drops it, and criteria across the flow and case suites shell out - # to $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py, which then exit 2 with - # "No such file or directory" and score as ordinary misses. Mounting is not - # loading: skills still come only from agent.plugins[].path below. - # - # The destination must be the HOST path, because $SKILLS_REPO_PATH is - # forwarded with its host value and the criteria interpolate it in-container. - # Writing that portably needs destination-side $VAR expansion, i.e. - # coder_eval > 0.11.1 (UiPath/coder_eval#128). This is the only line in this - # file that needs it. + # Repo root. Narrowing plugins.path to preview/ drops the automatic + # repo-root mount, and criteria shell out to + # $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py. Destination must be the + # host path, so this line needs UiPath/coder_eval#128. - $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro agent: permission_mode: acceptEdits allowed_tools: ["Skill", "Bash", "Read", "Write", "Edit", "Glob", "Grep"] plugins: - # PREVIEW ONLY. This path is BOTH the skill catalog and an automatic :ro - # bind mount at the same absolute path in-container, with $VAR expanded. + # PREVIEW ONLY. Also the source of the automatic repo-path bind mount. - type: "local" path: "$SKILLS_REPO_PATH/preview" ignore_patterns: [] post_run: - # A per-task `npm install` of the SDK leaves node_modules behind in every - # artifact dir; unpruned, a full Maestro run costs gigabytes of run output. + # Unpruned node_modules costs gigabytes across a full Maestro run. - command: "find . -maxdepth 5 -type d \\( -name node_modules -o -name .npm-prefix -o -name .venv \\) -prune -exec rm -rf {} +" timeout: 30 diff --git a/tests/experiments/same-ground-headtohead.yaml b/tests/experiments/same-ground-headtohead.yaml index 21061c04bf..892659cec8 100644 --- a/tests/experiments/same-ground-headtohead.yaml +++ b/tests/experiments/same-ground-headtohead.yaml @@ -28,10 +28,8 @@ defaults: - SKILLS_REPO_PATH - UIPATH_CLI_DISABLE_VERSION_SYNC extra_mounts: - # Same login destination nightly.yaml uses. This used to be a literal - # /home/tmatup/... because coder-eval forwards the host HOME and there is - # no destination-side $VAR expansion; the nightly path needs neither, so - # the hardcode is gone. Deeper target wins, so .skills still shadows. + # Same login destination nightly.yaml uses. Deeper target wins, so + # .skills still shadows. - ${SG_UIPATH_HOME}:/.uipath:rw - ${SG_EMPTY_SKILLS}:/.uipath/.skills:ro From 4fdc30837c44633fd0bc95c2019659c6e0e46e48 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 18:01:01 -0700 Subject: [PATCH 4/9] refactor(tests): drop flow-v2-preflight.sh for a one-line login check The script's only load-bearing assertion was that the `/.uipath` login mount resolves under the forwarded host HOME. That is one `docker run`, now in tests/README.md. The rest either duplicated what a failing run already reports or pinned CLI verbs that drift. Co-Authored-By: Claude Opus 5 (1M context) --- tests/README.md | 10 +++-- tests/docker/flow-v2-preflight.sh | 52 -------------------------- tests/experiments/flow-v2-preview.yaml | 3 +- 3 files changed, 8 insertions(+), 57 deletions(-) delete mode 100755 tests/docker/flow-v2-preflight.sh diff --git a/tests/README.md b/tests/README.md index 2eae26c7c5..44ae935fa1 100644 --- a/tests/README.md +++ b/tests/README.md @@ -196,9 +196,13 @@ the same name, so a run measures the Flow v2 authoring path rather than a mix of both generations. Narrowing `plugins.path` to `preview/` drops the automatic repo-root bind mount, so the root is remounted explicitly; the image also needs runtime npm auth for the `@uipath` scope. Login state mounts at `/.uipath`, -identical to `nightly.yaml`. Run `tests/docker/flow-v2-preflight.sh` first: it -asserts those preconditions in one container, each standing for a failure that -otherwise scores as a capability problem rather than a config one. +identical to `nightly.yaml`. Confirm that mount resolves before a full run, or +every tenant call fails as a capability problem rather than a config one: + +```bash +docker run --rm --env HOME="$HOME" -v ~/.uipath:/.uipath:rw \ + --entrypoint bash skills-codex:latest -c 'uip login status' +``` `activation.yaml` is a different shape from the tiered configs above — it runs the agent against single-prompt rows to measure whether the right skill fires (precision/recall/F1 per skill). Rows get a small turn budget (`max_turns: 3`) with `stop_early: true`: the armed `skill_triggered` criteria (`stop_when: auto`) end a row as soon as its outcome is live-decided. A positive row pass-stops the moment the expected skill engages; a negative row fail-stops on its first engagement. A wrong-skill engagement alone does NOT end a positive row — fail-stop is deferred while the row's positive criterion is still undecided, so a positive row that only misfires runs to the cap, as do rows with no engagement. Decided rows cost ~1 turn and a late-but-correct invocation is no longer truncated. Requires coder_eval >= 0.9.1. It's an opt-in benchmark, not a smoke gate. See [`tasks/activation/README.md`](tasks/activation/README.md). diff --git a/tests/docker/flow-v2-preflight.sh b/tests/docker/flow-v2-preflight.sh deleted file mode 100755 index 03f3aa7986..0000000000 --- a/tests/docker/flow-v2-preflight.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# Config gate for a docker run of tests/experiments/flow-v2-preview.yaml. -# -# Asserts only the things that, when wrong, score as capability failures -# instead of config errors. Product behaviour (flow check/compile/scaffold) is -# deliberately NOT checked here: the eval measures that, and asserting CLI -# verbs here just rots. -# -# Pass --env HOME with the HOST value, as the runner does. Without it a manual -# `docker run` authenticates against the image's /root and reproduces nothing. -# -# Usage: tests/docker/flow-v2-preflight.sh [image] [uipath-home] -# Requires NODE_AUTH_TOKEN (GitHub Packages read:packages, SSO-authorized). -set -uo pipefail - -IMG="${1:-skills-codex:latest}" -UIPATH_HOME="${2:-$HOME/.uipath}" -SKILLS_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -: "${NODE_AUTH_TOKEN:?NODE_AUTH_TOKEN must be set (GitHub Packages token)}" -[ -d "$UIPATH_HOME" ] || { echo "FAIL: no uipath home at $UIPATH_HOME"; exit 1; } - -LOG="$(mktemp)" -trap 'rm -f "$LOG"' EXIT - -docker run --rm \ - --env HOME="$HOME" \ - --env NODE_AUTH_TOKEN \ - --env UIPATH_CLI_DISABLE_VERSION_SYNC=1 \ - -v "$UIPATH_HOME:/.uipath:rw" \ - -v "$SKILLS_REPO:$SKILLS_REPO:ro" \ - --entrypoint bash "$IMG" -c ' - echo "HOME=$HOME" - echo "userconfig=$(npm config get userconfig)" - echo "uipath_registry=$(npm config get @uipath:registry)" - echo "token_seen=$([ -n "${NODE_AUTH_TOKEN:-}" ] && echo yes || echo no)" - uip login status 2>&1 | head -12 - d=$(mktemp -d); cd "$d"; npm init -y >/dev/null 2>&1 - npm install @uipath/flow-sdk >/dev/null 2>&1 - echo "SDK_VERSION=$(node -e "console.log(require(\"@uipath/flow-sdk/package.json\").version)" 2>&1)" - ' >"$LOG" 2>&1 - -fail=0 -chk() { if grep -qE "$2" "$LOG"; then echo " PASS $1"; else echo " FAIL $1"; fail=1; fi; } -echo "=== pre-flight: $IMG ===" -chk "container HOME is the forwarded host HOME" "^HOME=$HOME\$" -chk "npm userconfig is HOME-independent" '^userconfig=/root/\.npmrc$' -chk "@uipath scope resolves to GitHub Packages" '^uipath_registry=https://npm\.pkg\.github\.com/?$' -chk "GH Packages token reached the container" '^token_seen=yes$' -chk "uip reports a live login" '"Status": "Logged in"' -chk "flow-sdk installs in-sandbox" '^SDK_VERSION=[0-9]+\.[0-9]+\.[0-9]+$' -if [ "$fail" -ne 0 ]; then echo "GATE=FAIL"; echo "--- container output ---"; cat "$LOG"; exit 1; fi -echo "GATE=PASS" diff --git a/tests/experiments/flow-v2-preview.yaml b/tests/experiments/flow-v2-preview.yaml index 1e8097008b..7a7612e483 100644 --- a/tests/experiments/flow-v2-preview.yaml +++ b/tests/experiments/flow-v2-preview.yaml @@ -4,8 +4,7 @@ description: >- (uipath-maestro-{flow,case,bpmn}) as the ONLY skill catalog, under the docker driver. Shadows the shipped v1 skills of the same name by pointing the catalog at preview/ alone, so a run measures the SDK authoring path rather than a mix - of both generations. Pair with `tests/docker/flow-v2-preflight.sh`, which - asserts the preconditions this file depends on before a full run is launched. + of both generations. defaults: run_limits: From e4f42cd041457a3868e870a6813e3dc3944aaa13 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 18:15:51 -0700 Subject: [PATCH 5/9] refactor(tests): move preview npm auth out of the shared image into pre_run The npmrc layer set ENV NPM_CONFIG_USERCONFIG on tests/docker/Dockerfile, which changes npm resolution for every suite, not just the Flow v2 preview. Several skills tell agents to `npm install -g @uipath/cli`, which is public-npm-only for stable releases and carries no registry override, so a global @uipath -> GitHub Packages mapping is a hazard the preview run has no business creating. Revert the Dockerfile and write the npmrc from the experiment's pre_run instead, into $HOME, which is where npm resolves userconfig, so it holds whatever HOME the runner forwards and needs no NPM_CONFIG_USERCONFIG at all. pre_run executes inside the sandbox before the agent and fails the task loudly on error, so a broken write cannot masquerade as a capability miss. ${NODE_AUTH_TOKEN} stays literal; npm expands it at read time. Only flow-v2-preview.yaml needs this. No other experiment installs an @uipath package in-sandbox. Co-Authored-By: Claude Opus 5 (1M context) --- tests/docker/Dockerfile | 15 --------------- tests/experiments/flow-v2-preview.yaml | 18 +++++++++++++++++- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/tests/docker/Dockerfile b/tests/docker/Dockerfile index 6d14547303..1d1067191b 100644 --- a/tests/docker/Dockerfile +++ b/tests/docker/Dockerfile @@ -137,18 +137,3 @@ ENV DOTNET_ROOT=/usr/share/dotnet \ # loops in the turn timeline). uipath-langchain is declared in the # SimpleCodedAgent fixture's pyproject.toml but has no pre-built venv. RUN pip install --no-cache-dir "uipath-langchain>=0.9.26" - -# Runtime npm auth for the @uipath scope, needed by any task that runs -# `npm install @uipath/...` in-sandbox (the Maestro builder SDK, -# `@uipath/flow-sdk`, is published only to GitHub Packages). -# -# Token-LESS: npm expands ${NODE_AUTH_TOKEN} at read time, so nothing secret -# enters the image. NPM_CONFIG_USERCONFIG is required, not belt-and-braces: -# npm resolves userconfig to $HOME/.npmrc, and the runner forwards --env HOME -# with the HOST value, so npm would otherwise look in a dir the container does -# not have and fall through to the public registry, where the package 404s. -RUN printf '%s\n' \ - '@uipath:registry=https://npm.pkg.github.com/' \ - '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' \ - > /root/.npmrc -ENV NPM_CONFIG_USERCONFIG=/root/.npmrc diff --git a/tests/experiments/flow-v2-preview.yaml b/tests/experiments/flow-v2-preview.yaml index 7a7612e483..698d66e465 100644 --- a/tests/experiments/flow-v2-preview.yaml +++ b/tests/experiments/flow-v2-preview.yaml @@ -15,7 +15,6 @@ defaults: sandbox: driver: docker docker: - # Build with tests/docker/Dockerfile: its npmrc layer is required. image: skills-codex:latest network: bridge env_passthrough_extra: @@ -50,6 +49,23 @@ defaults: path: "$SKILLS_REPO_PATH/preview" ignore_patterns: [] + pre_run: + # Runtime npm auth for the @uipath scope. The shared image ships none: its + # build-time npmrc carries a literal token and is deleted in the same layer. + # @uipath/flow-sdk is published only to GitHub Packages, so without this + # every in-sandbox `npm install @uipath/flow-sdk` 404s against the public + # registry and each compile fails with it. Kept here rather than in + # tests/docker/Dockerfile so npm resolution is unchanged for every other + # suite. Written to $HOME, which is where npm resolves userconfig, so it + # holds whatever HOME the runner forwards. ${NODE_AUTH_TOKEN} stays literal: + # npm expands it at read time, so no token is written to disk. + - command: |- + mkdir -p "$HOME" && printf '%s\n' \ + '@uipath:registry=https://npm.pkg.github.com/' \ + '//npm.pkg.github.com/:_authToken=${NODE_AUTH_TOKEN}' \ + > "$HOME/.npmrc" + timeout: 30 + post_run: # Unpruned node_modules costs gigabytes across a full Maestro run. - command: "find . -maxdepth 5 -type d \\( -name node_modules -o -name .npm-prefix -o -name .venv \\) -prune -exec rm -rf {} +" From 93f2850370d50c0a07a40a97644027105d1c8aa5 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 21:09:16 -0700 Subject: [PATCH 6/9] ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval anthropic 1.0.0 (2026-08-20) moved its HTTP layer to httpx2. coder-eval 0.10.2 imports httpx in judge_bedrock without declaring it, so it only ever resolved transitively through anthropic 0.x. On a fresh install today llm_judge fails to import and every task carrying an llm_judge criterion errors at setup with score 0.00. Reproduced against the released wheel, nothing branch-specific: uv pip install "coder-eval==0.10.2" -> anthropic 1.0.0, no httpx -> import coder_eval.criteria.llm_judge ModuleNotFoundError: No module named 'httpx' uv pip install "coder-eval==0.10.2" "anthropic<1.0" -> anthropic 0.125.0, httpx 0.28.1, llm_judge imports coder_eval main already switched to httpx2, so this cap comes off with the next .coder-eval-version bump. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/smoke-rpa-skills.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/smoke-rpa-skills.yml b/.github/workflows/smoke-rpa-skills.yml index 96ec1c6473..ed6928bc5a 100644 --- a/.github/workflows/smoke-rpa-skills.yml +++ b/.github/workflows/smoke-rpa-skills.yml @@ -126,9 +126,12 @@ jobs: # Host coder-eval CLI = pinned wheel (coder_eval feed + ml-packages for # deps). Windows RPA tasks run under the tempdir driver, so no agent image. + # anthropic 1.0.0 (2026-08-20) swapped httpx for httpx2. The pinned + # coder-eval imports httpx without declaring it, so llm_judge stops + # importing and every task carrying one errors at setup with score 0. - name: Install coder-eval shell: bash - run: uv pip install --system "coder-eval==${{ steps.ceref.outputs.version }}" + run: uv pip install --system "coder-eval==${{ steps.ceref.outputs.version }}" "anthropic<1.0" - name: Configure NuGet feed for Helm packages shell: bash From 01ac115ee9d9062987059de4d837d5781a1483ce Mon Sep 17 00:00:00 2001 From: Bai Li Date: Fri, 21 Aug 2026 21:18:39 -0700 Subject: [PATCH 7/9] revert: "ci(rpa-smoke): cap anthropic below 1.0 for the pinned coder-eval" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 93f285037. The forward fix is already open as #2707, which moves the pin to 0.11.1 — the coder_eval release carrying beeceddc, "bump anthropic to 1.0.0, migrate Bedrock judge path to httpx2" — and exempts anthropic from the safe-chain package-age gate in the same four workflows. Keeping the cap would break that merge rather than help it: 0.11.1 declares anthropic>=1.0.0,<2.0.0, so the two constraints are unsatisfiable and the install step would fail outright. uv pip install "coder-eval==0.11.1" "anthropic<1.0" -> No solution found when resolving dependencies The cap also would not have turned the check green. With httpx restored the tasks reached the agent and then hit a second, unrelated Windows break: claude-agent-sdk 0.2.144 cannot find a native claude.exe, so all three crash with agent_crash after 3 attempts. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/smoke-rpa-skills.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/.github/workflows/smoke-rpa-skills.yml b/.github/workflows/smoke-rpa-skills.yml index ed6928bc5a..96ec1c6473 100644 --- a/.github/workflows/smoke-rpa-skills.yml +++ b/.github/workflows/smoke-rpa-skills.yml @@ -126,12 +126,9 @@ jobs: # Host coder-eval CLI = pinned wheel (coder_eval feed + ml-packages for # deps). Windows RPA tasks run under the tempdir driver, so no agent image. - # anthropic 1.0.0 (2026-08-20) swapped httpx for httpx2. The pinned - # coder-eval imports httpx without declaring it, so llm_judge stops - # importing and every task carrying one errors at setup with score 0. - name: Install coder-eval shell: bash - run: uv pip install --system "coder-eval==${{ steps.ceref.outputs.version }}" "anthropic<1.0" + run: uv pip install --system "coder-eval==${{ steps.ceref.outputs.version }}" - name: Configure NuGet feed for Helm packages shell: bash From 702927a3fc25caedeb52a37fa96ae3f2441bba24 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Mon, 24 Aug 2026 11:10:47 -0700 Subject: [PATCH 8/9] chore(tests): bump the coder-eval pin to 0.11.2 UiPath/coder_eval#128 shipped in 0.11.2, so `_validate_extra_mount` now expands `~` and `$VAR` in a destination and the flow-v2-preview repo-root mount loads instead of aborting. Carries 0.11.0's breaking changes into the whole suite: the cap-drop anti-cheat window on every `docker run`, and directory-only `task.reference`. The ~298 `reference: {file:}` tasks under uipath-troubleshoot are skipped with a migration error until #2707 lands their migration. Co-Authored-By: Claude Opus 5 (1M context) --- tests/.coder-eval-version | 2 +- tests/experiments/flow-v2-preview.yaml | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/.coder-eval-version b/tests/.coder-eval-version index 5eef0f10e8..bc859cbd6d 100644 --- a/tests/.coder-eval-version +++ b/tests/.coder-eval-version @@ -1 +1 @@ -0.10.2 +0.11.2 diff --git a/tests/experiments/flow-v2-preview.yaml b/tests/experiments/flow-v2-preview.yaml index 698d66e465..6b9dbf9006 100644 --- a/tests/experiments/flow-v2-preview.yaml +++ b/tests/experiments/flow-v2-preview.yaml @@ -37,7 +37,8 @@ defaults: # Repo root. Narrowing plugins.path to preview/ drops the automatic # repo-root mount, and criteria shell out to # $SKILLS_REPO_PATH/tests/tasks/**/_shared/*.py. Destination must be the - # host path, so this line needs UiPath/coder_eval#128. + # host path, which the framework only expands from 0.11.2 on + # (UiPath/coder_eval#128); earlier versions reject it at load. - $SKILLS_REPO_PATH:$SKILLS_REPO_PATH:ro agent: From f51d19a81b0616abff9c27174ebee2a55407c800 Mon Sep 17 00:00:00 2001 From: Bai Li Date: Mon, 24 Aug 2026 12:00:19 -0700 Subject: [PATCH 9/9] ci(smoke): exclude claude-agent-sdk 0.2.144 on the Windows RPA runner 0.2.144 is the first release in months published without a win_amd64 wheel, so uv falls back to the sdist and nothing bundles claude.exe. All three RPA smoke tasks crash at agent_crash before doing any work. coder-eval requires claude-agent-sdk>=0.2.124 with no upper bound, so exclude that single release rather than capping the range: 0.2.143 has the wheel, and a later release that restores it resolves normally. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/smoke-rpa-skills.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/workflows/smoke-rpa-skills.yml b/.github/workflows/smoke-rpa-skills.yml index 39d8f64f56..8c3bef47c4 100644 --- a/.github/workflows/smoke-rpa-skills.yml +++ b/.github/workflows/smoke-rpa-skills.yml @@ -126,9 +126,18 @@ jobs: # Host coder-eval CLI = pinned wheel (coder_eval feed + ml-packages for # deps). Windows RPA tasks run under the tempdir driver, so no agent image. + # + # claude-agent-sdk 0.2.144 shipped without a win_amd64 wheel, so uv falls + # back to the sdist and nothing bundles claude.exe — every task here then + # dies at agent_crash ("Claude Code not found"). coder-eval asks for + # >=0.2.124 with no upper bound, so exclude that single release instead of + # capping: a later version that restores the wheel resolves normally. - name: Install coder-eval shell: bash - run: uv pip install --system "coder-eval==${{ steps.ceref.outputs.version }}" + run: | + uv pip install --system \ + "coder-eval==${{ steps.ceref.outputs.version }}" \ + "claude-agent-sdk!=0.2.144" - name: Configure NuGet feed for Helm packages shell: bash