fix(tests): three false-negative guardrail failures from the 2026-09-10 nightlies #3202
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,178 @@ | ||
| #!/usr/bin/env python3 | ||
| """Selective `uip` shim for the BYOG guardrail tasks. | ||
|
|
||
| Shared by `coded/guardrails/byog_middleware`, `coded/guardrails/byog_decorator` | ||
| and `lowcode/guardrails/byog_pinning` -- one payload, so a CLI field rename is a | ||
| one-file edit and the three tasks cannot drift apart. Serves the two read-only | ||
| discovery paths an agent walks before it can pin a bring-your-own guardrail, | ||
| both answering with the configuration the prompts name (`byog-smoke-agent-pin`, | ||
| pii_detection): | ||
|
|
||
| * `uip agent guardrails list [--byo]` -> GuardrailDefinitionsList | ||
| * `uip guardrails byo-configurations list` -> ByoGuardrailConfigurationsList | ||
|
|
||
| Without `--byo`, `agent guardrails list` also returns the BUILT-IN | ||
| `pii_detection` entry sharing the same `Validator` name, so the agent must | ||
| disambiguate via `IsByo`/`ByoValidatorName` exactly as the skill teaches | ||
| rather than grabbing the first PII row. The two views agree on ids and | ||
| connector metadata, so cross-checking one against the other is consistent. | ||
|
|
||
| Every other invocation exec's the REAL `uip` found later on PATH, and | ||
| `--help` is handed to the real CLI so flag questions get real answers | ||
| (a canned payload there reads as "that flag does not exist"). | ||
|
|
||
| Why mock: the graded behavior is code wiring in `graph.py`, and both | ||
| checkers are pure AST analysis that never touch a tenant -- only the | ||
| agent's DISCOVERY step needed one. It could not keep depending on the | ||
| shared smoke tenant. `byo-configurations create` probes the connection | ||
| server-side with no skip flag, so a configuration cannot be seeded | ||
| per-run, and the tenant has no guardrail-capable connection to bind | ||
| (same reasoning as the byog_pinning mock, which fixed the low-code half | ||
| of this family). Left live, these tasks were graded on whether an | ||
| external tenant happened to hold a fixture: on 2026-08-24 the tenant | ||
| held three configurations, none of them `byog-smoke-agent-pin`; by | ||
| 2026-09-10 it held none at all, and both discovery calls returned | ||
| `Data: []`. The agent then correctly refused to fabricate a validator | ||
| name and changed nothing, scoring 0.00 -- a fixture outage rendered as a | ||
| skill regression. | ||
|
|
||
| `mock_path_dirs: [mocks]` PATH-prepends this script's directory inside the | ||
| sandbox. Linux smoke only (no `windows` tag on any of the three tasks), hence no | ||
| `uip.cmd` twin -- same as the ixp and platform guardrails mocks. | ||
| """ | ||
|
|
||
| import json | ||
| import os | ||
| import shutil | ||
| import sys | ||
|
|
||
| MOCK_DIR = os.path.dirname(os.path.abspath(__file__)) | ||
|
|
||
| VALIDATOR_NAME = "byog-smoke-agent-pin" | ||
| CONNECTION_ID = "18fb337c-29b7-4162-a9e8-0c05b01cf4df" | ||
| CONFIGURATION_ID = "e5723bb8-fbc2-4317-c7d7-08de803bc010" | ||
| CONNECTOR_NAME = "Azure AI Content Safety" | ||
| CONNECTOR_KEY = "uipath-azure-contentsafety" | ||
|
|
||
| PII_PARAMETERS = [ | ||
| { | ||
| "Type": "enum-list", | ||
| "Id": "entities", | ||
| "Required": True, | ||
| "DefaultValue": ["Email", "PhoneNumber"], | ||
| "Options": [ | ||
| "Email", | ||
| "PhoneNumber", | ||
| "Person", | ||
| "Address", | ||
| "USSocialSecurityNumber", | ||
| ], | ||
| "DisplayName": "Entities", | ||
| "Description": "PII entity types to detect.", | ||
| }, | ||
| { | ||
| "Type": "map-enum", | ||
| "Id": "entityThresholds", | ||
| "Required": True, | ||
| "DefaultValue": {"Email": 0.5, "PhoneNumber": 0.5}, | ||
| "KeySource": "entities", | ||
| "Min": 0, | ||
| "Max": 1, | ||
| "DisplayName": "Per-entity threshold", | ||
| "Description": "Confidence threshold (0-1) per selected entity.", | ||
| }, | ||
| ] | ||
|
|
||
| BYO_ENTRY = { | ||
| "Validator": "pii_detection", | ||
| "IsByo": True, | ||
| "Status": "Available", | ||
| "AllowedScopes": ["Agent", "Llm", "Tool"], | ||
| "GuardrailStages": { | ||
| "Agent": ["PreExecution", "PostExecution"], | ||
| "Llm": ["PreExecution", "PostExecution"], | ||
| "Tool": ["PreExecution", "PostExecution"], | ||
| }, | ||
| "DisplayName": "PII Detection", | ||
| "Description": ( | ||
| "Detects personally identifiable information. Served by the " | ||
| "tenant's external bring-your-own guardrail provider." | ||
| ), | ||
| "ByoValidatorName": VALIDATOR_NAME, | ||
| "ByoConnectionId": CONNECTION_ID, | ||
| "ByoConfigurationId": CONFIGURATION_ID, | ||
| "ByoConnectorName": CONNECTOR_NAME, | ||
| "ByoConnectorKey": CONNECTOR_KEY, | ||
| "FolderKey": "627fe423-5c73-464a-abff-41fdaad6ac19", | ||
| "Parameters": PII_PARAMETERS, | ||
| } | ||
|
|
||
| BUILTIN_ENTRY = { | ||
| "Validator": "pii_detection", | ||
| "IsByo": False, | ||
| "Status": "Available", | ||
| "AllowedScopes": ["Agent", "Llm", "Tool"], | ||
| "GuardrailStages": { | ||
| "Agent": ["PreExecution", "PostExecution"], | ||
| "Llm": ["PreExecution", "PostExecution"], | ||
| "Tool": ["PreExecution", "PostExecution"], | ||
| }, | ||
| "DisplayName": "PII Detection", | ||
| "Description": ( | ||
| "Detects personally identifiable information using Azure " | ||
| "Cognitive Services (UiPath built-in)." | ||
| ), | ||
| "Parameters": PII_PARAMETERS, | ||
| } | ||
|
|
||
| # The admin-side view of the same record. Field names mirror the CLI's own | ||
| # documented examples for `guardrails byo-configurations`, so the fields the | ||
| # skill teaches are the fields that come back. | ||
| CONFIGURATION = { | ||
| "Id": CONFIGURATION_ID, | ||
| "ConnectionId": CONNECTION_ID, | ||
| "ValidatorName": VALIDATOR_NAME, | ||
| "ValidatorType": "pii_detection", | ||
| "FallbackOnUiPath": False, | ||
| "Enabled": True, | ||
| "CreatedAt": "2026-07-02T11:40:00Z", | ||
| "UpdatedAt": None, | ||
| "ConnectorKey": CONNECTOR_KEY, | ||
| "ConnectorName": CONNECTOR_NAME, | ||
| "ConnectionName": f"{CONNECTOR_NAME} - Prod", | ||
| "ValidConnection": True, | ||
| } | ||
|
|
||
|
|
||
| def real_uip(): | ||
| path = os.environ.get("PATH", "") | ||
| parts = [p for p in path.split(os.pathsep) if p and os.path.abspath(p) != MOCK_DIR] | ||
| return shutil.which("uip", path=os.pathsep.join(parts)) | ||
|
|
||
|
|
||
| def emit(code, data): | ||
| print(json.dumps({"Result": "Success", "Code": code, "Data": data}, indent=2)) | ||
| return 0 | ||
|
|
||
|
|
||
| def main(): | ||
| args = sys.argv[1:] | ||
| literal = [a for a in args if not a.startswith("-")] | ||
|
|
||
| # A help request is a question about flags, not about tenant data. | ||
| if "--help" not in args and "-h" not in args: | ||
| if literal[:3] == ["agent", "guardrails", "list"]: | ||
| data = [BYO_ENTRY] if "--byo" in args else [BUILTIN_ENTRY, BYO_ENTRY] | ||
| return emit("GuardrailDefinitionsList", data) | ||
| if literal[:3] == ["guardrails", "byo-configurations", "list"]: | ||
| return emit("ByoGuardrailConfigurationsList", [CONFIGURATION]) | ||
|
|
||
| real = real_uip() | ||
| if not real: | ||
| sys.stderr.write("uip (shim): real uip CLI not found on PATH\n") | ||
| return 127 | ||
| os.execv(real, [real] + args) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| sys.exit(main()) | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 minor — second copy of this payload, and the twin is now the weaker one.
tests/tasks/uipath-agents/lowcode/guardrails/byog_pinning/mock_template/mocks/uipcarries the sameBYO_ENTRYandBUILTIN_ENTRY, the same GUIDs, the same validator name. This file adds two things it does not have:guardrails byo-configurations list, so a low-code agent that follows theByoConfigurationIdcross-reference atlowcode/.../guardrails.md:642still falls through to the real CLI and getsData: [], the exact fixture-outage failure this PR is fixing on the coded half.--helppassthrough, sobyog_pinningstill answers a flag question with a canned payload.Two copies that already disagree will keep disagreeing, and a CLI field rename now needs both edited. Suggest pointing
byog_pinning.yamlat this fixture and deleting its local copy (it already reaches across with../_fixtures/..., though I have not verified the harness accepts a path that climbs out oflowcode/), or hoisting the shim to a sharedtests/tasks/uipath-agents/_fixtures/.Not blocking, and either way this file is the better of the two.