[CI Failure Doctor] CI Failure Investigation - Run #35833

[github-actions]

🏥 CI Failure Investigation - Run #35833

Summary

Unit and integration tests for pkg/workflow started failing after the latest push because the sandbox runtime (SRT) configuration no longer passes validation. The frontmatter parser/schema now only accepts "default"/"awf" for the top-level sandbox string and rejects the previously supported sandbox-runtime/srt options, so the tests cannot even compile the workflows they exercise.

Failure Details

• Run: 22028425564
• Commit: 9c7d4f4
• Trigger: push

Root Cause Analysis

The sandbox extraction logic (extractSandboxConfig/isSupportedSandboxType) and the JSON schema only allow the legacy string values "default" or "awf" now. Any frontmatter that still uses sandbox: sandbox-runtime (or the sandbox.agent.id: srt shorthand) is dropped because isSupportedSandboxType rejects sandbox-runtime, so SandboxConfig is not populated and the compiler falls back to AWF defaults. The tests that enabled SRT now fail during validation and the new strict-mode checks (strict_mode_skips_validation_when_SRT_is_enabled, TestStrictModeFirewallValidation) complain that the firewall must remain enabled. Other sandbox tests such as TestSandboxRuntimeFeatureFlagRequired, TestCustomAWFCommandExecution/custom_command_and_args_for_SRT, and TestIsSandboxEnabled now fail during compilation because the schema rejects the sandbox-runtime string/object that their frontmatter uses.

Failed Jobs and Errors

• test: multiple sandbox-related unit tests fail, e.g., TestStrictModeFirewallValidation reports "strict mode: firewall must be enabled for copilot engine with network restrictions" and TestCustomAWFCommandExecution shows "at '/sandbox': got object, want string" when parsing sandbox-runtime configs.
• Integration: Workflow Runtime & Setup: the pipeline fails because TestSandboxRuntimeFeatureFlagViaEnv/TestSandboxRuntimeFeatureFlagRequired frontmatter now violates schema (value must be one of 'default' / 'awf') and compilation aborts before running the integration.
• Integration: Workflow Misc Part 2: sandbox-agent tests (TestSandboxAgentEnablesDefaultTools, TestIsSandboxEnabled) report that isSandboxEnabled returns false when legacy srt configurations are requested, leading to compilation failures before any assertions are reached.

Investigation Findings

• Running go test ./pkg/workflow locally reproduces the same failures as the test job, confirming the break is deterministic and not infra-related.
• The .changeset/patch-remove-srt-support.md entry and the new schema confirm this is intentional—SRT is being retired—so the code now rejects the old sandbox-runtime shorthands.
• None of the sandbox/SRT tests were updated to the new configuration format (and some rely on sandbox-runtime being available), so they now hit the validation errors immediately.

Recommended Actions

• Update the sandbox/SRT tests and sample workflows to use the supported configuration path (e.g., sandbox.agent.id: awf/srt with sandbox-runtime feature flag) or remove the obsolete tests if SRT is no longer supported.
• If SRT should still be configurable, reintroduce the legacy sandbox-runtime string/object support in extractSandboxConfig/isSupportedSandboxType and the schema, or add a migration so existing workflows and tests keep compiling.

Prevention Strategies

Add a regression check whenever sandbox schema defaults change—run go test ./pkg/workflow and any sandbox-specific integration tests after modifying pkg/workflow/sandbox.go or the schema so we catch config-breaking changes before merging.

AI Team Self-Improvement

Before removing a deprecated configuration mode (e.g., sandbox-runtime), update or remove every test and workflow that exercises it and rerun go test ./pkg/workflow to verify the new defaults compile.

Historical Context

I reviewed the latest [CI Failure Doctor] issues (e.g., #15792, #15789, #15782, #15775, #15765); they all covered different failures and none mention this sandbox-runtime regression, so there is no duplicate investigation for run #35833.

🩺 Diagnosis provided by CI Failure Doctor

To install this workflow, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. View source at https://github.com/githubnext/agentics/tree/ea350161ad5dcc9624cf510f134c6a9e39a6f94d/workflows/ci-doctor.md.

• expires on Feb 16, 2026, 2:54 AM UTC

[github-actions]

Run #35834 reproduces the same sandbox regression: test failed because isSandboxEnabled still returns false for legacy sandbox.type configs, so the sandbox/strict-mode tests and TestCompilerIsSandboxEnabled abort before they ever reach the assertions. I’m updating isSandboxEnabled to treat a populated sandbox.type (e.g., the AWF legacy value) as enabling the sandbox and planned to verify with go test -run TestCompilerIsSandboxEnabled ./pkg/workflow, but the environment could not download Go 1.25 (proxy access is forbidden), so the run never executed locally yet.

Once the toolchain issue is resolved I’ll rerun the targeted tests and follow up here.

AI generated by CI Failure Doctor

[github-actions]

Run #35836 (https://github.com/github/gh-aw/actions/runs/22028606446, commit 6ea365e) continues to fail for the same sandbox-runtime regression described in this issue. The test job now reports the TestCompilerIsSandboxEnabled suite failing because isSandboxEnabled no longer recognizes legacy sandbox.type usage, and the Integration: Workflow Runtime & Setup job aborts before exercising the sandbox-runtime flags because the frontmatter schema rejects sandbox: sandbox-runtime (error: at '/sandbox': 'oneOf' failed, none matched). Please treat run 35836 as the latest occurrence of the same regression.

AI generated by CI Failure Doctor

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #15860

View newer issue

---

This action was performed automatically by the CI Failure Doctor workflow.