Skip to content

feat: migrate feature tests to devcontainer features test CLI + copilot-instructions - #11

Merged
C. Spencer Beggs (spencerbeggs) merged 19 commits into
mainfrom
copilot/declare-devcontainer-features
May 1, 2026
Merged

C. Spencer Beggs (spencerbeggs) merged 19 commits into
mainfrom
copilot/declare-devcontainer-features

Conversation

Copilot AI commented May 1, 2026 •

Copy link
Copy Markdown
Contributor

Replaces ad-hoc sudo bash install.sh && sudo bash test.sh CI testing with the official devcontainer features test CLI pattern from the Microsoft feature-starter reference. Adds structured scenario definitions, per-scenario assertion scripts, and a copilot-instructions.md with commit conventions.

Test pattern migration

scenarios.json — converted from a custom metadata array (unused by tooling) to the CLI-native { scenarioName: devcontainer.json-fragment } object format. Each key is a named scenario that maps to a real devcontainer.json fragment:

{
  "explicit_pnpm": {
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
    "features": {
      "node": {},
      "package-manager": { "packageManager": "pnpm@10.20.0" }
    }
  }
}

test.sh — all 8 features converted from raw bash assertions to check/reportResults from dev-container-features-test-lib (set -e, not set -euo pipefail — the test lib uses unset variables internally).

5 new per-scenario assertion scripts — install_custom_version.sh (biome), explicit_pnpm.sh + explicit_yarn.sh (package-manager), explicit_version.sh (zig), remote_user_vscode.sh (homebrew, exercises the _REMOTE_USER install path).

CI & tooling

  • test.yml — discovery step delegates to .github/scripts/discover-features.js (ESM); test step runs devcontainer features test -f <id> -i mcr.microsoft.com/devcontainers/base:ubuntu [--skip-autogenerated] --skip-duplicated . with ::group:: log collapsing; after each run, .github/scripts/parse-test-results.js parses the TEST REPORT section into /tmp/test-results.json ({ passed, failed, total, allPassed }); step summary renders a per-scenario | Scenario | Result | table from the JSON.
  • publish.yml — same test + parse pattern; Save result artifact now includes the per-feature *-results.json; summarize step renders a | Feature | Scenario | Result | table.
  • test-feature.yml — workflow_dispatch single-feature run now also parses results and fails the job when any scenario fails.
  • scripts/test-feature.sh — drops the act dependency; delegates to devcontainer features test, captures output, calls parse-test-results.js, and exits non-zero on scenario failures so pnpm run test:feature <id> reliably signals failure locally.
  • scripts/validate-feature.sh — structural checks now enforce source dev-container-features-test-lib, reportResults, object-format scenarios.json (rejects null/primitives/arrays), presence of each scenario's .sh file, and set -e specifically (rejects set -euo pipefail).

New scripts

  • .github/scripts/parse-test-results.js — ESM script; reads a devcontainer features test log (default /tmp/test.log), extracts Passed/Failed scenario names from the TEST REPORT block, and writes a structured JSON results file. Accepts optional [logPath] [outPath] arguments.
  • .github/scripts/discover-features.js — ESM script; builds the { id, skipAutogenerated } matrix used by test.yml's discover step.

Feature installer fixes

  • src/zig/install.sh — reads ZIGVERSION (the env var the devcontainer CLI injects for the zigVersion option) with ZIG_VERSION as a fallback, so scenario overrides like zigVersion: "0.13.0" now take effect.
  • src/homebrew/install.sh — guards _REMOTE_USER with id -u before using it as BREW_USER; falls back to creating/using the linuxbrew user with an [INFO] log if the account is absent from the image.
  • src/claude-code/install.sh — replaces the deprecated npm install -g @anthropic-ai/claude-code with the official native installer (curl -fsSL https://claude.ai/install.sh | bash). Node.js/npm is no longer required. The installer now runs as _REMOTE_USER (via su -) when a non-root remote user exists in the image — the same pattern as the homebrew feature — so the binary lands in the user's home directory and is accessible at runtime. Falls back to a world-readable shared prefix (/usr/local/lib/claude-code with chmod -R a+rX) when no non-root remote user is available. A symlink in /usr/local/bin/claude provides system-wide PATH access in both cases. Version bumped 0.1.0 → 0.2.0, version option removed (not supported by the native installer), and the non-standard install.shell field replaced with the correct entrypoint field.

Autogenerated test fix

  • test/homebrew/test.sh — the autogenerated scenario test previously checked su - linuxbrew to verify Homebrew, but mcr.microsoft.com/devcontainers/base:ubuntu always sets _REMOTE_USER=vscode. When _REMOTE_USER=vscode, install.sh runs the Homebrew installer as the vscode user and never creates the linuxbrew account, causing su - linuxbrew to fail. Updated to use the full binary path directly (matching the pattern in remote_user_vscode.sh).

Documentation

  • test-patterns.md — rewritten to cover the three-file pattern, check/reportResults semantics, all CLI modes, local testing, and the --skip-autogenerated convention.
  • FEATURE.instructions.md — atomic update rule now includes scenario .sh files; five-file rule updated to require matching scenario scripts.
  • .github/copilot-instructions.md — new file documenting the @savvy-web/commitlint Silk preset: allowed types, format rules, body restrictions, and DCO sign-off requirement.
  • docs/features/claude-code.md — updated to document the native installer, remove the deprecated version option, and add a concrete usage snippet.

Copilot AI and others added 5 commits May 1, 2026 19:42
- rust: chown RUSTUP_HOME + CARGO_HOME to _REMOTE_USER after install
- homebrew: use _REMOTE_USER as Linux install user when non-root, falling
  back to dedicated linuxbrew account; update test to mirror logic
- node: export shared COREPACK_HOME=/usr/local/share/corepack in profile.d
- package-manager: export same COREPACK_HOME before corepack calls
- zig: remove spurious sudo calls (feature scripts already run as root)
- fix pre-existing missing execute bit on node + package-manager scripts
- bump all changed features to 0.2.0

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/8c66a117-fe5e-463a-a782-560068c2d637

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>
…de bits

- Revert src/node/install.sh, src/package-manager/install.sh,
  test/node/test.sh, test/package-manager/test.sh from 100755 back to
  100644 — shell scripts are not stored with the execute bit in git
- CLAUDE.md: remove "← must be executable" annotation; add "Executable Bits"
  section explaining the Husky/CI convention and that agents must not chmod +x
- SHELL.instructions.md: add explicit "Executable Bits" rule for agents
- FEATURE.instructions.md: add execute-bit note to Five-File Completeness Rule
- validate-feature.sh: change executable-bit checks from hard [FAIL] to [INFO]
- devcontainer SKILL.md: replace "Missing execute permission" common mistake
  with the correct explanation of the Husky/CI convention
- devcontainer references/test-patterns.md: clarify executable bit is managed
  by Husky/CI, not committed to git
- debug-feature SKILL.md: remove "Missing execute permission" category;
  clarify chmod note refers to installed binaries, not source scripts
- debug-feature references/common-failure-modes.md: rewrite §4 to distinguish
  between installed-binary permissions and the source script convention
- new-feature SKILL.md: remove "marked executable" from checklist

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/aaec7de0-c396-4a94-b485-b78cdc2a69a7

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>
…nd SKILL.md

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/fd30aafa-d106-4810-8a11-6be9a7342c6d

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Migrates this repo’s devcontainer feature testing from ad-hoc bash/CI execution to the official devcontainer features test CLI pattern, with structured scenarios and standardized assertion/reporting. Also updates several feature installers to better support non-root devcontainer users and adds repository-wide instructions (including Copilot commit conventions).

Changes:

  • Convert feature tests to dev-container-features-test-lib (check + reportResults) and switch scenarios.json to the CLI-native object format with per-scenario <name>.sh assertion scripts.
  • Update CI/workflows and local scripts to run devcontainer features test (including --skip-autogenerated for features depending on other local features).
  • Improve feature installers for devcontainer user ergonomics (e.g., Rust ownership transfer, shared Corepack cache, Homebrew remote user installs) and refresh internal documentation/instructions.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/zig/test.sh Migrates Zig default test to check/reportResults.
test/zig/scenarios.json Converts Zig scenarios to CLI object format (explicit version scenario).
test/zig/explicit_version.sh Adds Zig per-scenario assertions for non-default version.
test/rust/test.sh Migrates Rust default test to check/reportResults and expands assertions.
test/rust/scenarios.json Switches Rust scenarios to empty object (no scenarios).
test/package-manager/test.sh Migrates package-manager default test to test-lib (auto-gen intended to be skipped).
test/package-manager/scenarios.json Adds CLI scenarios wiring in node dependency for pnpm/yarn cases.
test/package-manager/explicit_yarn.sh Adds yarn scenario assertion script.
test/package-manager/explicit_pnpm.sh Adds pnpm scenario assertion script.
test/node/test.sh Migrates Node default test to check/reportResults.
test/node/scenarios.json Switches Node scenarios to empty object (no scenarios).
test/homebrew/test.sh Migrates Homebrew default test to check/reportResults.
test/homebrew/scenarios.json Adds scenario to exercise remoteUser=vscode.
test/homebrew/remote_user_vscode.sh Adds Homebrew per-scenario assertions for non-root remote user installs.
test/claude-code/test.sh Migrates Claude Code default test to check/reportResults.
test/claude-code/scenarios.json Switches Claude Code scenarios to empty object (no scenarios).
test/biome/test.sh Migrates Biome default test to check/reportResults.
test/biome/scenarios.json Adds Biome scenario for custom version install.
test/biome/install_custom_version.sh Adds Biome per-scenario assertions for custom version.
test/act/test.sh Migrates act default test to check/reportResults.
test/act/scenarios.json Switches act scenarios to empty object (no scenarios).
src/zig/install.sh Removes unnecessary sudo and relies on root feature execution.
src/zig/devcontainer-feature.json Bumps Zig feature version.
src/rust/install.sh Transfers Rust toolchain ownership to _REMOTE_USER when appropriate.
src/rust/devcontainer-feature.json Bumps Rust feature version.
src/package-manager/install.sh Uses shared COREPACK_HOME for cross-user reuse of prepared package managers.
src/package-manager/devcontainer-feature.json Bumps package-manager feature version.
src/node/install.sh Exports shared COREPACK_HOME via profile and ensures directory exists.
src/node/devcontainer-feature.json Bumps Node feature version.
src/homebrew/install.sh Installs Homebrew on Linux as _REMOTE_USER when non-root, otherwise via linuxbrew.
src/homebrew/devcontainer-feature.json Bumps Homebrew feature version.
scripts/validate-feature.sh Updates validation for test-lib usage, set -e, scenarios object format, and scenario script presence.
scripts/test-feature.sh Removes act dependency; runs devcontainer features test directly.
CLAUDE.md Documents executable-bit convention and updates feature file rules accordingly.
.github/workflows/test.yml Replaces install/test bash steps with devcontainer features test matrix execution.
.github/workflows/test-feature.yml Updates local workflow_dispatch testing to use devcontainer features test.
.github/workflows/publish.yml Updates publish pipeline’s test step to use devcontainer features test.
.github/skills/new-feature/SKILL.md Updates checklist regarding executable-bit convention.
.github/skills/devcontainer/references/test-patterns.md Rewrites testing reference to the 3-file CLI pattern and new rules.
.github/skills/devcontainer/references/feature-anatomy.md Documents _REMOTE_USER/_CONTAINER_USER env vars and canonical guard pattern.
.github/skills/devcontainer/SKILL.md Updates best practices and notes (remote user ownership, executable bits).
.github/skills/debug-feature/references/common-failure-modes.md Clarifies execute-permission failures for installed binaries vs repo scripts.
.github/skills/debug-feature/SKILL.md Updates failure mode list and guidance to match new conventions.
.github/instructions/SHELL.instructions.md Adds explicit guidance on executable-bit convention.
.github/instructions/FEATURE.instructions.md Updates version-bump + completeness rules for scenario scripts and executable bits.
.github/copilot-instructions.md Adds Copilot-facing commit/commitlint + DCO conventions.

Comment thread src/zig/install.sh
Comment on lines +59 to +63
# Install (feature scripts run as root; sudo is not needed)
rm -rf "$ZIG_ROOT"
mkdir -p "$ZIG_ROOT"
cp -r "$ZIG_EXTRACTED"/* "$ZIG_ROOT"/
ln -sf "$ZIG_BIN" /usr/local/bin/zig

Copilot AI May 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zigVersion option from devcontainer-feature.json will be injected as ZIGVERSION, but install.sh only reads ZIG_VERSION. This means scenario installs (e.g. zigVersion=0.13.0) will be ignored and the scenario test will fail (and users can’t override the version). Align the env var name (or add a fallback) so feature options actually take effect.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot apply changes based on this feedback

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already applied in the branch. src/zig/install.sh reads ZIG_VERSION="${ZIGVERSION:-${ZIG_VERSION:-0.12.0}}" (commit 3be3543) — ZIGVERSION (the CLI-injected form) takes precedence, with ZIG_VERSION as a fallback.

Comment thread src/homebrew/install.sh
Comment on lines +39 to +44
REMOTE_USER="${_REMOTE_USER:-}"
if [[ -n "$REMOTE_USER" && "$REMOTE_USER" != "root" ]]; then
BREW_USER="$REMOTE_USER"
else
NONINTERACTIVE=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
BREW_USER="linuxbrew"
useradd -m -s /bin/bash "$BREW_USER" 2>/dev/null || true

Copilot AI May 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When _REMOTE_USER is set to a non-root user, this script immediately uses it for chown/su without verifying the account exists. If remoteUser is configured to a user that isn’t present yet in the image, the feature install will fail due to set -euo pipefail. Consider guarding with id -u "$REMOTE_USER" (and falling back to creating/using linuxbrew) before assigning BREW_USER.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot apply changes based on this feedback

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already applied in the branch (commit 3be3543). The _REMOTE_USER path now calls id -u "$REMOTE_USER" >/dev/null 2>&1 before assigning BREW_USER; if the account doesn't exist the script logs [INFO] and falls back to creating/using the linuxbrew user.

Comment thread scripts/validate-feature.sh Outdated
@spencerbeggs

Copy link
Copy Markdown
Contributor

Copilot apply changes based on the comments in this thread

I also noticed that our test workflow had a one scenario that passed and one scenario that failed but we marked the job as all passed: https://github.com/savvy-web/devcontainers/actions/runs/25231483946/job/73987845521?pr=11

We need to collect the reports from each scenarios and fail the jobs overall, but continue so they report so we have an accurate understanding of what parts of tests fail

Copilot AI and others added 2 commits May 1, 2026 20:35
…ario null check, CI failure detection

The devcontainer CLI injects zigVersion as ZIGVERSION (no separator).
Read ZIGVERSION first with ZIG_VERSION as a fallback.

Guard _REMOTE_USER in homebrew install with id -u existence check
before assigning as BREW_USER; fall back to linuxbrew user with
a warning if the account is absent from the image.

Tighten scenarios.json structural check in validate-feature.sh to
reject null, numbers, and strings in addition to arrays.

Detect per-scenario failures in test.yml and publish.yml by grepping
the captured devcontainer features test output for the ❌ Failed:
marker emitted in the TEST REPORT, since the CLI exits 0 even when
individual scenarios fail.

Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/45c46e47-d95b-4aac-a3c2-9555f010454b

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>
Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/45c46e47-d95b-4aac-a3c2-9555f010454b

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI commented May 1, 2026

Copy link
Copy Markdown
Contributor Author

Copilot apply changes based on the comments in this thread

I also noticed that our test workflow had a one scenar...

Applied all three review fixes and the CI scenario-failure issue in commit 3be3543 (with a minor follow-up in 58c0ca8):

  • zig env var — install.sh now reads ZIGVERSION (the name the CLI injects) first, with ZIG_VERSION as a fallback, so zigVersion: "0.13.0" in a scenario is actually honoured instead of silently ignored.
  • homebrew remote user guard — the _REMOTE_USER path now calls id -u "$REMOTE_USER" before assigning it as BREW_USER; if the account doesn't exist the script logs [INFO] and falls back to creating/using the linuxbrew user rather than failing under set -euo pipefail.
  • validate-feature.sh null check — the Node assertion now checks !j || typeof j !== 'object' || Array.isArray(j) so null, numbers, and strings are rejected as well as arrays.
  • CI scenario failures — devcontainer features test exits 0 even when individual scenarios fail. Both test.yml and publish.yml now grep the captured log for the ❌ Failed: marker emitted in the TEST REPORT and exit 1 when any are found, so steps.test.outcome becomes failure and the job accurately reflects reality while still running to completion (the continue-on-error: true + summary pattern is preserved).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

Comment thread scripts/validate-feature.sh Outdated
Comment thread .github/workflows/test-feature.yml Outdated
chmod +x "$TEST_SCRIPT"
sudo bash "$TEST_SCRIPT"

devcontainer features test "${ARGS[@]}" .

Copilot AI May 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow runs devcontainer features test but does not fail the job when individual scenarios fail. In test.yml/publish.yml you already note the CLI can exit 0 even on scenario failures and you grep the TEST REPORT to force a non-zero exit. Apply the same failure detection here so workflow_dispatch runs reliably signal failure.

Copilot uses AI. Check for mistakes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot apply changes based on this feedback

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c1b8094. Two changes:

  • test-feature.yml — the CLI invocation is now wrapped with set +e / set -e and captures PIPESTATUS[0] as TEST_EXIT_CODE, so parse-test-results.js always runs even when the CLI exits non-zero. The final exit combines the CLI exit code and results.allPassed.

  • parse-test-results.js — when no TEST REPORT section is found in the log (CLI crash, output format change), allPassed is now set to false rather than silently returning an empty-but-passing result.

Comment thread scripts/test-feature.sh Outdated
@spencerbeggs

Copy link
Copy Markdown
Contributor

Copilot apply changes based on the comments in this thread

Is a better way to parse the test sucesss and failure scenarios? We are searching the log for emojis, but could we out put a JSON file with just the scenarios and expected outcomes and results?

…ction, fix review issues

Add .github/scripts/parse-test-results.js (ESM) to parse the devcontainer
features test TEST REPORT into { passed, failed, total, allPassed } JSON.
Add .github/scripts/discover-features.js (ESM) to build the test matrix,
replacing a 25-line inline node block in test.yml.

All three workflow test steps and scripts/test-feature.sh now call the
script directly instead of embedding inline node -e parse logic. Simple
one-liner require() checks in node -e remain unchanged (node -e always
runs in CJS mode regardless of package.json type field).

Also applies review fixes:
- validate-feature.sh: tighten set -e check to reject set -euo pipefail;
  fix regex to not require trailing whitespace after pipefail; tighten
  scenarios.json structural check to reject null/primitive values
- test-feature.yml: add scenario failure detection via parse-test-results.js
- scripts/test-feature.sh: capture output, parse JSON results, propagate
  failures; fix fallback JSON to include total field
- publish.yml: fix fallback JSON in Save result step to include total field;
  expand summarize table to show per-scenario rows from results JSON
- src/zig/install.sh: read ZIGVERSION (CLI-injected) with ZIG_VERSION fallback
- src/homebrew/install.sh: guard _REMOTE_USER with id -u existence check

Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/5f032a45-357b-4921-89c6-104e4436b192

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI commented May 1, 2026

Copy link
Copy Markdown
Contributor Author

Copilot apply changes based on the comments in this thread

Is a better way to parse the test sucesss and failure ...

Landed in f116aa3. Here's what changed:

JSON-based failure detection — instead of grepping for emoji characters in the log, a new .github/scripts/parse-test-results.js script reads the TEST REPORT section and writes /tmp/test-results.json with the shape { passed, failed, total, allPassed }. All three workflow test steps and scripts/test-feature.sh call that script directly. Failure detection is then a simple one-liner: node -e "process.exit(require('fs').readFileSync('/tmp/test-results.json')... .allPassed ? 0 : 1)".

On the --input-type=module question: node -e always runs in CommonJS mode regardless of "type": "module" in package.json, so require() works fine for simple inline one-liners without any flag. The new .js files use ESM naturally because of the "type": "module" setting — no flag needed there either.

The three review thread items are also applied:

  • validate-feature.sh set -e check now specifically rejects set -euo pipefail
  • test-feature.yml propagates scenario failures via the JSON
  • scripts/test-feature.sh captures output and exits non-zero on scenario failures

The discover step in test.yml was also extracted to .github/scripts/discover-features.js to keep the workflow clean.

Signed-off-by: C. Spencer Beggs <spencer@savvyweb.systems>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 5 comments.

Comment on lines +26 to +30
const idx = log.indexOf("TEST REPORT");
const section = idx >= 0 ? log.slice(idx) : log;
const passed = [...section.matchAll(/Passed:\s+'([^']+)'/g)].map((m) => m[1]);
const failed = [...section.matchAll(/Failed:\s+'([^']+)'/g)].map((m) => m[1]);

Copilot AI May 1, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the log doesn't contain a "TEST REPORT" section (CLI crash or output format change), passed/failed will be empty and the computed result effectively becomes a silent success. Consider explicitly detecting the missing report and setting allPassed=false (and/or recording a parse error) so callers don't treat "no parsed results" as passing.

Copilot uses AI. Check for mistakes.
Comment thread .github/workflows/test.yml Outdated
Comment thread .github/workflows/test-feature.yml Outdated
Comment thread .github/workflows/publish.yml Outdated
Comment thread src/biome/install.sh Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: C. Spencer Beggs <spencer@beg.gs>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: C. Spencer Beggs <spencer@beg.gs>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: C. Spencer Beggs <spencer@beg.gs>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: C. Spencer Beggs <spencer@beg.gs>
test-feature.yml: wrap the devcontainer CLI invocation with set +e /
set -e and capture PIPESTATUS so parse-test-results.js always runs even
when the CLI exits non-zero. The final exit is based on both the CLI
exit code and results.allPassed.

parse-test-results.js: when no TEST REPORT section is found in the log
(CLI crash, output format change) set allPassed to false instead of
silently returning an empty-but-passing result set.

Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/b087a77e-2fc9-40e9-aa13-edf76ede352b

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>
…und fix

Signed-off-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI commented May 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Copilot apply changes based on the comments in this thread

I also noticed that our test workflow had a one scenar...

All three review fixes and the CI scenario-failure issue were already applied in earlier commits (3be3543, bc764c4, c1b8094). The current branch has the zig env var fallback, homebrew id -u guard, scenarios.json null/primitive rejection, and JSON-based failure detection with set +e + PIPESTATUS across all three workflow files.

Copilot apply changes based on the comments in this thread

Is a better way to parse the test sucesss and failure ...

Already implemented in bc764c4. .github/scripts/parse-test-results.js reads the TEST REPORT section and writes /tmp/test-results.json with the shape { passed: string[], failed: string[], total: number, allPassed: boolean }. All three workflow test steps and scripts/test-feature.sh call that script — no emoji-grepping anywhere. A missing TEST REPORT section (CLI crash) now sets allPassed: false rather than silently passing.

@spencerbeggs

Copy link
Copy Markdown
Contributor

Copilot If you look at the latest test runs were are having issues only with the claude-code and homebrew installations. I am not clear on why homebrew is failing, but our claude-code installation is trying to use node. We really should use the native install method. Please read the docs and update the feature to use the native method: https://code.claude.com/docs/en/overview.md

Copilot AI and others added 2 commits May 1, 2026 21:44
…en test

claude-code: Replace deprecated npm install with official native installer
(curl -fsSL https://claude.ai/install.sh | bash). Node.js/npm is no longer
required. The version option is removed since the native installer always
installs the latest release. The entrypoint field format is corrected from
the non-standard install.shell to entrypoint. Version bumped 0.1.0 -> 0.2.0.

homebrew: The autogenerated test assumed _REMOTE_USER=root and checked for a
linuxbrew user via su. The base image (mcr.microsoft.com/devcontainers/base:ubuntu)
always sets _REMOTE_USER=vscode, so the linuxbrew user is never created and
su - linuxbrew fails. Updated test.sh to use the full binary path directly,
matching the pattern already used in remote_user_vscode.sh.

Signed-off-by: Copilot <copilot@github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/b1a37736-11b0-45bc-819f-cee3aa03e0a9

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>
Signed-off-by: Copilot <copilot@github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/b1a37736-11b0-45bc-819f-cee3aa03e0a9

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI commented May 1, 2026 •

Copy link
Copy Markdown
Contributor Author

Copilot If you look at the latest test runs were are having issues only with the claude-code and homebrew installations. I am not clear on why homebrew is failing, but our claude-code installation is...

Fixed in commit 4343930 (with supporting changes in 9cf8e07).

claude-code (src/claude-code/install.sh) — replaced the deprecated npm install with the official native installer: curl -fsSL https://claude.ai/install.sh | bash. Node.js/npm is no longer required. The feature also now uses the correct entrypoint field format and removes the no-longer-relevant version option. Version bumped 0.1.0 → 0.2.0.

homebrew (test/homebrew/test.sh) — the autogenerated test was checking su - linuxbrew but mcr.microsoft.com/devcontainers/base:ubuntu always sets _REMOTE_USER=vscode, so install.sh runs Homebrew as the vscode user and the linuxbrew user is never created. Updated to use the full binary path directly (same pattern as remote_user_vscode.sh), which matches what actually happens at runtime.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • anthropic.com
    • Triggering command: /usr/bin/curl curl -fsSL REDACTED (dns block)
  • claude.ai
    • Triggering command: /usr/bin/curl curl -fsSL REDACTED -collect-logs (dns block)
  • code.claude.com
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
    • Triggering command: /usr/bin/curl curl -fsSL REDACTED (dns block)
  • docs.anthropic.com
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
  • https://api.github.com/repos/anthropics/claude-code/releases/latest
    • Triggering command: /usr/bin/curl curl -fsSL REDACTED (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Signed-off-by: C. Spencer Beggs <spencer@savvyweb.systems>
@spencerbeggs

Copy link
Copy Markdown
Contributor

Copilot Almost finished. We are still having issues with the claude-code native install. It appears to install locally, but there is an issue with the path. I see this warning in the install docker build:

[2026-05-01T22:18:03.917Z]
#13 [dev_containers_target_stage 5/5] RUN --mount=type=bind,from=dev_containers_feature_content_source,source=claude-code_0,target=/tmp/build-features-src/claude-code_0 cp -ar /tmp/build-features-src/claude-code_0 /tmp/dev-container-features && chmod -R 0755 /tmp/dev-container-features/claude-code_0 && cd /tmp/dev-container-features/claude-code_0 && chmod +x ./devcontainer-features-install.sh && ./devcontainer-features-install.sh && rm -rf /tmp/dev-container-features/claude-code_0
#13 0.138 ===========================================================================
#13 0.138 Feature : Claude Code Agent
#13 0.138 Description : Installs the Claude Code CLI agent globally via the official native installer (curl | bash). Does not require Node.js or npm.
#13 0.138 Id : Unknown
#13 0.138 Version : 0.2.0

[2026-05-01T22:18:03.917Z] #13 0.138 Documentation : https://github.com/savvy-web/devcontainers/blob/main/docs/features/claude-code.md
#13 0.138 Options :
#13 0.138
#13 0.138 ===========================================================================

[2026-05-01T22:18:04.070Z] #13 0.141 [INFO] Installing Claude Code CLI via native installer...

[2026-05-01T22:18:05.409Z] #13 1.630 Setting up Claude Code...

[2026-05-01T22:18:05.669Z] #13 1.890
#13 1.890 Checking installation status...
[2026-05-01T22:18:05.844Z]
#13 1.914 Installing Claude Code native build latest...
[2026-05-01T22:18:07.795Z]
#13 4.016 Setting up launcher and shell integration...
[2026-05-01T22:18:07.974Z]
#13 4.044 ⚠ Setup notes:
#13 4.044 ● Native installation exists but ~/.local/bin is not in your PATH. Run:
#13 4.044
#13 4.044 echo 'export PATH="$HOME/.local/bin:$PATH"' >> your shell config file && source your shell config file

[2026-05-01T22:18:09.823Z] #13 6.044
#13 6.044 ✔ Claude Code successfully installed!
#13 6.044
#13 6.044 Version: 2.1.126
#13 6.044
#13 6.044 Location: ~/.local/bin/claude
#13 6.044
#13 6.044
[2026-05-01T22:18:09.823Z]
#13 6.044 Next: Run claude --help to get started
#13 6.044
#13 6.044 ⚠ Setup notes:
#13 6.044 ● Native installation exists but ~/.local/bin is not in your PATH. Run:
#13 6.044
#13 6.044 echo 'export PATH="$HOME/.local/bin:$PATH"' >> your shell config file && source your shell config file

[2026-05-01T22:18:11.823Z] #13 8.044

[2026-05-01T22:18:11.975Z] #13 8.112
#13 8.112 ✅ Installation complete!
#13 8.112
#13 8.196 2.1.126 (Claude Code)

The native installer writes the binary to $HOME/.local/bin/claude.
When running as root during the Docker build ($HOME=/root, mode 700),
the installed binary is inaccessible to non-root container users
(e.g. vscode) causing `claude: command not found` at test time.

Follow the same pattern as the homebrew feature: use _REMOTE_USER
when it is non-root and present in the image, running the installer
via `su - $INSTALL_USER` so the binary lands in that user's home
directory and is owned/accessible by them at runtime. Fall back to a
world-accessible shared prefix (/usr/local/lib/claude-code) with
chmod -R a+rX when no non-root remote user is available.

Signed-off-by: Copilot <copilot@github.com>

Agent-Logs-Url: https://github.com/savvy-web/devcontainers/sessions/4eba7a96-ba2b-4125-ade0-bd06444cd2c4

Co-authored-by: spencerbeggs <109114+spencerbeggs@users.noreply.github.com>

Copilot AI commented May 1, 2026

Copy link
Copy Markdown
Contributor Author

Copilot Almost finished. We are still having issues with the claude-code native install. It appears to install locally, but there is an issue with the path. I see this warning in the install docker b...

Fixed in c05a5e7. The root cause was that the native installer places the binary at $HOME/.local/bin/claude, and since the feature install runs as root ($HOME=/root, mode 700), the installed binary was inaccessible to the vscode container user at test time — hence claude: command not found.

The fix follows the same pattern as homebrew/install.sh: read _REMOTE_USER, and if it is non-root and present in the image, run the installer via su - $INSTALL_USER so everything installs under that user's home directory. Since the test container runs as vscode, the binary and all launcher-referenced files are fully owned/accessible by the right user. A symlink from /usr/local/bin/claude makes claude available on PATH for all users.

For the fallback (no non-root remote user), HOME is overridden to /usr/local/lib/claude-code with chmod -R a+rX for world-accessibility.

@spencerbeggs
C. Spencer Beggs (spencerbeggs) merged commit 28c6c1a into main May 1, 2026
26 of 28 checks passed
@spencerbeggs
C. Spencer Beggs (spencerbeggs) deleted the copilot/declare-devcontainer-features branch May 1, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants