Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,23 @@ permissions:
packages: read

jobs:
nestedvirt-tools:
runs-on: ubuntu-latest
timeout-minutes: 10
Comment on lines 35 to +38

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This workflow runs on both push and pull_request, but has no workflow-level concurrency group. Each update can therefore leave an older run executing alongside the newer one, duplicating the fixture checks and the other integration jobs; add a group keyed by workflow and ref (with cancel-in-progress: true if superseded runs are not needed).

Suggestion:

Suggested change
jobs:
nestedvirt-tools:
runs-on: ubuntu-latest
timeout-minutes: 10
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
nestedvirt-tools:
runs-on: ubuntu-latest
timeout-minutes: 10

steps:
Comment on lines +36 to +39

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

other · low
This job has no timeout-minutes, so a hung fixture subprocess or server can consume the runner until GitHub's much longer default timeout. Please set an explicit bounded timeout appropriate for these syntax/fixture checks.

Suggestion:

Suggested change
nestedvirt-tools:
runs-on: ubuntu-latest
steps:
nestedvirt-tools:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added timeout-minutes: 10 to nestedvirt-tools in a436054. I checked the actual prior job: it succeeded in 7 seconds, and no hang was observed. The 10-minute limit is a bounded resource-protection measure with ample allowance for checkout and Python setup, replacing the 360-minute default for this lightweight job only. The existing per-request/readiness/process timeouts remain unchanged.

Comment on lines +36 to +39

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security · medium
This job only checks files and runs the fixture script, but it inherits the workflow-wide packages: read permission (and checkout persists the token in git config). Since these commands execute repository-controlled code, granting package-read access is unnecessary exposure; declare job-level permissions with only contents: read (or move the package permission to the jobs that pull from GHCR).

Suggestion:

Suggested change
nestedvirt-tools:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
nestedvirt-tools:
permissions:
contents: read
runs-on: ubuntu-latest
timeout-minutes: 10
steps:

- uses: actions/checkout@v6
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- name: Check nested KVM smoke syntax
run: |
python3 -m py_compile scripts/tests/nestedvirt-smoke.py
sh -n scripts/tests/nestedvirt-guest.sh
- name: Verify real fixture rebuild
run: |
python3 scripts/tests/nestedvirt-smoke.py check-fixture
python3 -O scripts/tests/nestedvirt-smoke.py check-fixture
Comment on lines +48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
This new CI job only compiles the scripts and runs check-fixture; there is no workflow step invoking the run subcommand (and no other caller exists in the changed workflow tree). Consequently the nested-KVM API matrix—KVM availability, QEMU startup, fork, pause/resume, and snapshot restore—is never exercised in CI. Please add a hardware-capable job that prepares the kernel/config/API key/node and runs the end-to-end command, or avoid presenting this as nested-KVM integration coverage.

Suggestion:

Suggested change
- name: Verify real fixture rebuild
run: |
python3 scripts/tests/nestedvirt-smoke.py check-fixture
python3 -O scripts/tests/nestedvirt-smoke.py check-fixture
- name: Run nested KVM smoke matrix
run: |
python3 scripts/tests/nestedvirt-smoke.py run ...

Comment on lines +48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
This job only exercises check-fixture; there is no invocation of the run subcommand anywhere in the workflow, so the newly added template build, nested-KVM boot, fork, pause/resume, and Firecracker paths are not covered by CI. A syntax/fixture-only pass can therefore leave the main smoke test broken. Please either add the required KVM/kernel/API setup and run the full command here, or wire it into a separate CI job.

Comment on lines +48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
This job never invokes the run subcommand, so the newly added runtime path is not exercised in CI: fixture boot contents, QEMU/KVM startup, fork/restore, pause/resume, and cleanup can all regress while these checks remain green. Add a hardware-capable job/step with the required node endpoint, API key, inner kernel, and runtime prerequisites, or explicitly wire this smoke test into the existing integration job.

Suggestion:

Suggested change
- name: Verify real fixture rebuild
run: |
python3 scripts/tests/nestedvirt-smoke.py check-fixture
python3 -O scripts/tests/nestedvirt-smoke.py check-fixture
- name: Run nested KVM smoke
run: |
python3 scripts/tests/nestedvirt-smoke.py run "$AENV_ENDPOINT" "$AENV_API_KEY_FILE" "$RUNNER_TEMP/nestedvirt" --inner-kernel "$INNER_KERNEL" --kernel-release "$KERNEL_RELEASE"

Comment on lines +48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
This job only runs syntax checks and check-fixture; it never invokes the run command, so the API/template-build, nested-KVM, fork, and pause/resume paths added here are not exercised by CI. Please add this smoke to a KVM-capable job (or explicitly wire it into the existing E2E setup) so those regressions cannot merge unnoticed.


agentenv-tests:
runs-on: ubuntu-latest
steps:
Expand Down
8 changes: 4 additions & 4 deletions config/deps_manifest.toml
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
[firecracker.kvm]
version = "1.15.1-patch-v1"
url = "https://github.com/kvcache-ai/firecracker/releases/download/aenv-deps/firecracker-{version}-{arch}.tgz"
version = "1.15.1-patch-v2"
url = "https://github.com/kvcache-ai/firecracker/releases/download/v{version}/firecracker-{version}-{arch}.tgz"

[firecracker.pvm]
version = "v1.17.0-next.1"
url = "https://github.com/kvcache-ai/firecracker-next/releases/download/{version}/firecracker-next-{version}-{arch}.tgz"

[kernel.kvm]
version = "vmlinux-6.1.175"
url = "https://github.com/kvcache-ai/firecracker/releases/download/aenv-deps/{version}-{arch}"
version = "vmlinux-6.18"
url = "https://github.com/kvcache-ai/firecracker/releases/download/v1.15.1-patch-v2/{version}-{arch}"

[kernel.pvm]
version = "6.12.33-pvm"
Expand Down
75 changes: 75 additions & 0 deletions docs/src/internals/sandbox-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ This document describes the public sandbox API, the global `ConfigManager` under
- Each sandbox gets its own temporary work directory.
- `FirecrackerSandbox::resume_from_snapshot_config(&FirecrackerSnapshotConfig)`
- Creates a new sandbox handle and immediately starts it from a snapshot config.
- Uses the node's currently configured Firecracker, whether taking a prewarmed
process or starting a new one. Cross-version snapshot compatibility is
provided by that Firecracker runtime.
- Uses `vm_state_path`, the memory overlay image config, and
`base_rootfs_path` from the config.
- Returns a running sandbox after Firecracker and envd are ready.
Expand Down Expand Up @@ -563,6 +566,78 @@ client API and the binary that setup downloads:
cargo test -p agentenv --lib
```

## 4.2) Nested KVM Validation

`scripts/tests/nestedvirt-smoke.py` builds a test image and exercises template
restoration, fork, and pause/resume with QEMU running inside an x86_64 sandbox.
Run it when changing the nested-KVM runtime, snapshot handling, or guest kernel.

1. Use an x86_64 host with nested KVM and an outer guest kernel with built-in
KVM. Install Python 3.11+, QEMU x86 system binaries, static BusyBox at
`/bin/busybox`, cpio, and Bash. Provide a bootable bzImage for the inner guest
and a fresh work directory with at least 5 GiB free.
2. Prepare a dedicated AgentENV node using the host setup above. Use separate
`AENV_HOME_PATH`, `AENV_RUNTIME_PATH`, and network pools; select test assets
with `AENV_DEPS_PATH`. Bind the API to loopback and set
`[sandbox_proxy].domains`, for example to `["nested.local"]`.
3. Keep the complete `[firecracker].boot_args` string and append `no-kvmapf`;
the runner checks this argument. Set `[pool.block].enabled = true`.
Configure regctl for the runner's loopback registry with
`--host reg=127.0.0.1:18091,tls=disabled`; match the port to `--port`.
4. Start the node, then use `python3 scripts/tests/nestedvirt-smoke.py run --help`
for SDK installation, parameters, and a complete invocation. Pass the node's
TOML file with `--config`, including its effective boot arguments.
5. Require a successful exit and one result per selected phase and round in
`<work>/evidence/api-results.json` (nine with defaults). Guest commands and
output are in `api-commands.jsonl`. Stop the test node after the run.

The runner builds one template, then creates a source sandbox and two fork
children per case. The guest payload `nestedvirt-guest.sh` starts and stops QEMU with `accel=kvm` and supplies
an inner-guest counter.

| Phase | Recovery check |
|-------|----------------|
| `unused` | A fresh inner VM starts after fork and pause/resume. |
| `initialized` | QEMU created with `-S` retains KVM-enabled `prelaunch` state. |
| `running` | The inner counter continues advancing after fork and pause/resume. |

Each case also starts and stops a fresh inner VM in the source and both children;
the initialized case replaces its stopped inner VM. Existing snapshots retain
their captured kernel and boot arguments, so use fresh templates to test changes
to either. Hosted CI runs syntax and `check-fixture` checks; the lifecycle matrix
runs on nested-KVM hardware.

## 4.3) Legacy v9 Snapshot Recovery

`orchestrator_v9_snapshot_upgrade` in `tests/integration/orchestrator.rs` runs the
current AgentENV test binary with old and current Firecracker processes. It
saves v9 state, invalidates the recorded executable path, restores with the
configured runtime, checks envd, and saves again as v1000.

1. Prepare `v9-node.toml` and `current-node.toml` with their respective runtimes
and matching helpers. Use absolute paths and the same state, storage,
network, guest kernel, and boot arguments in both configurations.
2. Complete the integration host setup above and build the daemon with
`make build-ublk PROFILE=debug`.
3. Run the test explicitly; it is ignored by default because it needs both
runtime versions:

```bash
AENV_HOME_PATH=/absolute/path/isolated-test/home \
AENV_RUNTIME_PATH=/absolute/path/isolated-test/run \
AENV_DEPS_PATH=/absolute/path/test-deps \
AENV_UBLK_DAEMON_BINARY_PATH="$PWD/target/debug/uvm-ublk-daemon" \
AENV_CONFIG_PATH=/absolute/path/current-node.toml \
AENV_TEST_V9_CONFIG=/absolute/path/v9-node.toml \
scripts/run-with-capabilities.sh cargo test -p agentenv \
--test orchestrator_integration orchestrator::orchestrator_v9_snapshot_upgrade \
-- --exact --ignored --nocapture
```

For service-upgrade validation, create templates and paused records with the old
AgentENV binary, stop it, start the new binary against the same state, and verify
restore, fork, and another pause/resume cycle.

## 5) Running Custom Scripts

### Run commands via envd
Expand Down
45 changes: 45 additions & 0 deletions scripts/tests/nestedvirt-guest.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
#!/bin/sh
# Copyright 2026 AgentENV contributors.
# SPDX-License-Identifier: Apache-2.0
set -eu
mode=${1:-fixture-init}
if [ "$#" -gt 0 ]; then shift; fi
case "$mode" in
fixture-init)
while true; do /bin/busybox sleep 3600; done
Comment on lines +8 to +9

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · high
The outer fixture's /dev is created empty by prepare_fixture, and this init never mounts devtmpfs or creates /dev/kvm. Consequently the later test -c /dev/kvm check and QEMU's KVM accelerator cannot work in the built image unless the rootfs happens to contain the device node (it does not here). Mount devtmpfs in the outer init, or otherwise provision the KVM device before running the smoke matrix.

;;
inner-init)
/bin/busybox mount -t proc proc /proc
/bin/busybox mount -t sysfs sysfs /sys
n=0
while true; do
echo "INNER_TICK=$n"
n=$((n+1))
/bin/busybox sleep 1
done
;;
start-inner)
/bin/busybox ip link set lo up
exec /usr/bin/qemu-system-x86_64 -machine microvm,accel=kvm -cpu host \
-m 128 -smp 1 -nodefaults -no-reboot -nographic -serial stdio \
-qmp tcp:127.0.0.1:4444,server=on,wait=off \
-kernel /inner-bzImage -initrd /inner.cpio \
-append 'console=ttyS0 panic=1 rdinit=/init' "$@"
;;
stop-inner)
pid=$(/bin/busybox cat "${1:?PID_FILE}")
test "$(/bin/busybox readlink /proc/"$pid"/exe)" = /usr/bin/qemu-system-x86_64
kill -TERM "$pid"
attempt=0
while kill -0 "$pid" 2>/dev/null; do
processes=$(/bin/busybox ps -o pid,stat)
state=$(printf '%s\n' "$processes" | /bin/busybox awk -v target="$pid" '$1 == target {print $2}')
case "$state" in Z*|"") break ;; esac
attempt=$((attempt + 1))
test "$attempt" -lt 100
/bin/busybox sleep 0.1
done
echo "QEMU_STOPPED=$pid"
;;
*) echo "Unknown guest operation: $mode" >&2; exit 2 ;;
esac
Loading
Loading