Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions app/arcbox-api/src/system.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ use std::sync::Arc;
use arcbox_grpc::SystemService;
use arcbox_protocol::v1::{
Empty, ResolveContainerFsRequest, ResolveContainerFsResponse, ResolveImageFsRequest,
ResolveImageFsResponse, SetSystemVmBackendRequest, SetupStatus, SystemVmBackend,
SystemVmBackendInfo, VcpuDebug, VirtioDebugInfo, VirtioDeviceDebug, VirtioQueueDebug,
setup_status,
ResolveImageFsResponse, SandboxCapability, SetSystemVmBackendRequest, SetupStatus,
SystemVmBackend, SystemVmBackendInfo, VcpuDebug, VirtioDebugInfo, VirtioDeviceDebug,
VirtioQueueDebug, setup_status,
};
use tokio::sync::watch;
use tokio_stream::Stream;
Expand Down Expand Up @@ -265,6 +265,19 @@ impl SystemService for SystemServiceImpl {
}))
}

async fn get_sandbox_capability(
&self,
_request: Request<Empty>,
) -> Result<Response<SandboxCapability>, Status> {
let runtime = self.runtime.ready()?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Host Capability Waits For Guest

This host-only query uses shared_runtime, which is unavailable until full runtime initialization and System VM readiness. During daemon startup, clients receive UNAVAILABLE even after early_runtime contains the configured backend, so they cannot discover the capability without first booting the VM.

Context Used: AGENTS.md (source)

let cap = runtime.sandbox_capability();
Ok(Response::new(SandboxCapability {
supported: cap.supported,
reason: cap.reason,
backend: backend_to_proto(cap.backend) as i32,
}))
}

async fn get_virtio_debug(
&self,
_request: Request<Empty>,
Expand Down
31 changes: 31 additions & 0 deletions app/arcbox-cli/src/commands/sandbox.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,33 @@ async fn sandbox_channel() -> Result<Channel> {
})
}

/// Fails fast when the daemon reports that sandboxes cannot run on this host
/// or System VM backend, instead of booting a microVM that would land in
/// `failed` with an opaque KVM error. This is a host-side check — no round-trip
/// into the guest. Transport errors (e.g. an older daemon without the RPC) fall
/// through so the create still proceeds, where the guest agent remains the
/// backstop.
async fn ensure_sandbox_supported(channel: &Channel) -> Result<()> {
use arcbox_grpc::SystemServiceClient;

let mut client = SystemServiceClient::new(channel.clone());
match client
.get_sandbox_capability(tonic::Request::new(arcbox_protocol::v1::Empty {}))
.await
{
Ok(resp) => {
let cap = resp.into_inner();
if !cap.supported {
anyhow::bail!("{}", cap.reason);
}
Ok(())
}
// The capability RPC is unavailable (older daemon, not ready); let the
// create proceed rather than blocking on a missing pre-check.
Err(_) => Ok(()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Capability Errors Silently Bypass Check

This arm treats every gRPC status as an unavailable capability RPC. A permission, data-loss, or internal server error therefore starts sandbox creation instead of reporting the failed pre-check; only compatibility and temporary-availability statuses should fall through.

Suggested change
Err(_) => Ok(()),
Err(status)
if matches!(
status.code(),
tonic::Code::Unimplemented | tonic::Code::Unavailable
) =>
{
Ok(())
}
Err(status) => Err(status.into()),

}
}

/// Attaches the default `x-machine` metadata header to a tonic request for
/// daemon-side routing to the guest VM agent.
fn attach_machine<T>(mut request: tonic::Request<T>) -> tonic::Request<T> {
Expand Down Expand Up @@ -294,6 +321,10 @@ fn parse_labels(raw: &[String]) -> Result<HashMap<String, String>> {

async fn execute_create(args: CreateArgs) -> Result<()> {
let channel = sandbox_channel().await?;

// Reject unsupported hosts/backends up front with an actionable message.
ensure_sandbox_supported(&channel).await?;

let mut client = SandboxServiceClient::new(channel);

let labels = parse_labels(&args.label)?;
Expand Down
2 changes: 1 addition & 1 deletion app/arcbox-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ pub use macos::{
#[cfg(feature = "macos-ipsw-install")]
pub use macos::{PullPhase, PullSource};
pub use migration::MigrationManager;
pub use runtime::{Runtime, SandboxPortExposure};
pub use runtime::{Runtime, SandboxCapability, SandboxPortExposure};
pub use vm::{SharedDirConfig, VmConfig, VmManager};
pub use vm_lifecycle::{
ActivityScope, DEFAULT_MACHINE_NAME, DefaultVmConfig, HealthMonitor, VmLifecycleConfig,
Expand Down
62 changes: 62 additions & 0 deletions app/arcbox-core/src/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,49 @@ pub struct SandboxPortExposure {
pub guest_port: u16,
}

/// Whether this host and System VM backend can run sandboxes.
///
/// Produced by [`Runtime::sandbox_capability`].
#[derive(Debug, Clone)]
pub struct SandboxCapability {
/// Whether sandboxes are runnable on the current host and backend.
pub supported: bool,
/// Actionable reason when unsupported; empty when supported.
pub reason: String,
/// The System VM backend the capability was evaluated against.
pub backend: arcbox_vmm::VmBackend,
}

/// Pure decision behind [`Runtime::sandbox_capability`], split out for testing.
///
/// Host nested-virt support is the hard gate (M3+ / macOS 15+ hardware), so it
/// is checked first: on hardware that lacks it, switching backends cannot help.
/// Only when the hardware is capable does the HV backend become the actionable
/// blocker (nested virtualization is unavailable under Hypervisor.framework).
fn evaluate_sandbox_capability(
backend: arcbox_vmm::VmBackend,
host_nested_virt: bool,
) -> (bool, String) {
if !host_nested_virt {
return (
false,
"sandbox requires nested virtualization: the VZ backend on Apple \

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason strings here are macOS-specific ("the VZ backend on Apple Silicon M3 or newer with macOS 15+" and, below, "arcbox system backend vz"), but evaluate_sandbox_capability and the host_supports_nested_virt() probe both have real Linux paths (KVM nested module params). A Linux host with KVM nesting disabled would surface the Apple-Silicon message. Minor, and sandboxes are macOS-oriented, but the text is misleading off macOS.

Technical details
# macOS-centric reason strings on a cross-platform helper

## Affected sites
- `app/arcbox-core/src/runtime.rs:172-186` — both `reason` strings name Apple Silicon / macOS 15+ / `arcbox system backend vz`, yet the function is reached on Linux via `Runtime::sandbox_capability``arcbox_hypervisor::host_supports_nested_virt` (Linux KVM probe in `virt/arcbox-hypervisor/src/linux/mod.rs`).

## Required outcome
- On Linux, an unsupported result should read as a KVM-nesting message rather than an Apple-Silicon one — or the reason should be platform-conditional.

## Open questions for the human
- Is `arcbox sandbox create` a supported surface on Linux at all? If the System VM / sandbox architecture is macOS-only in practice, this is cosmetic and can be left as-is; if Linux is a real target, the strings should branch per platform.

Silicon M3 or newer with macOS 15+; this host does not support it"
.to_string(),
);
}
if backend == arcbox_vmm::VmBackend::Hv {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ignore the macOS backend selector when evaluating Linux hosts

On a Linux x86_64 host with KVM nesting enabled and vm.backend = "hv" (also accepted through ARCBOX_VM_BACKEND=hv), this rejects sandbox creation even though the Linux VMM always executes initialize_linux() and ignores the macOS-only VmBackend selector. The new Linux host probe therefore returns true for a usable nested-KVM setup, but this branch reports it unsupported and makes arcbox sandbox create bail before reaching the guest; apply the HV/VZ restriction only on macOS.

Useful? React with 👍 / 👎.

return (
false,
"sandbox requires nested virtualization, which is unavailable under \
the HV backend (current backend: HV); switch with \
`arcbox system backend vz`"
.to_string(),
);
}
(true, String::new())
}

impl Runtime {
/// Creates a new runtime with the given configuration.
///
Expand Down Expand Up @@ -415,6 +458,25 @@ impl Runtime {
self.vm_lifecycle.backend()
}

/// Reports whether this host and System VM backend can run sandboxes.
///
/// Sandboxes are nested Firecracker microVMs; they need `/dev/kvm` inside
/// the System VM, which the host exposes only with nested virtualization
/// enabled (VZ backend on Apple Silicon M3+ with macOS 15+). Computing this
/// on the host lets `arcbox sandbox create` and clients fail fast without a
/// round-trip into the guest.
#[must_use]
pub fn sandbox_capability(&self) -> SandboxCapability {
let backend = self.system_vm_backend();
let (supported, reason) =
evaluate_sandbox_capability(backend, arcbox_hypervisor::host_supports_nested_virt());
SandboxCapability {
supported,
reason,
backend,
}
}

/// Switches the System VM's hypervisor backend (HV <-> VZ) and restarts the
/// VM so it takes effect.
///
Expand Down
25 changes: 25 additions & 0 deletions app/arcbox-core/src/runtime/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -238,3 +238,28 @@ fn resolve_bind_ip_defaults_and_loopback() {
);
assert_eq!(super::resolve_bind_ip("not-an-ip"), None);
}

#[test]
fn sandbox_capability_gates_on_nested_virt_and_backend() {
use arcbox_vmm::VmBackend;

// VZ backend on nested-virt-capable hardware: sandboxes run.
let (supported, reason) = super::evaluate_sandbox_capability(VmBackend::Vz, true);
assert!(supported);
assert!(reason.is_empty());

// HV backend on capable hardware: actionable "switch to VZ" message.
let (supported, reason) = super::evaluate_sandbox_capability(VmBackend::Hv, true);
assert!(!supported);
assert!(reason.contains("HV"));
assert!(reason.contains("backend vz"));

// No host nested-virt support: hardware message regardless of backend,
// since switching backends cannot help.
for backend in [VmBackend::Vz, VmBackend::Hv] {
let (supported, reason) = super::evaluate_sandbox_capability(backend, false);
assert!(!supported);
assert!(reason.contains("nested virtualization"));
assert!(reason.contains("M3"));
}
}
16 changes: 16 additions & 0 deletions rpc/arcbox-protocol/proto/api.proto
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,12 @@ service SystemService {
// Gets the System VM's current hypervisor backend.
rpc GetSystemVmBackend(Empty) returns (SystemVmBackendInfo);

// Reports whether this host and System VM backend can run sandboxes
// (nested-virtualization microVMs). Lets clients pre-check and grey out
// sandbox features without booting a microVM that would fail with an
// opaque KVM error.
rpc GetSandboxCapability(Empty) returns (SandboxCapability);

// Switches the System VM's hypervisor backend (HV <-> VZ). The choice is
// persisted; the System VM is restarted so it takes effect, which stops
// running containers. Returns the resulting backend.
Expand Down Expand Up @@ -268,6 +274,16 @@ message ResolveImageFsResponse {
repeated string lower_dirs = 1;
}

// Whether sandboxes (nested-virtualization microVMs) can run on this host.
message SandboxCapability {
// Whether sandboxes are runnable on the current host and backend.
bool supported = 1;
// Actionable reason when unsupported; empty when supported.
string reason = 2;
// The System VM backend the capability was evaluated against.
SystemVmBackend backend = 3;
}

// Diagnostic snapshot of the System VM's virtio devices and vCPUs.
message VirtioDebugInfo {
repeated VirtioDeviceDebug devices = 1;
Expand Down
15 changes: 15 additions & 0 deletions rpc/arcbox-protocol/src/generated/arcbox.v1.rs
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,21 @@ pub struct ResolveImageFsResponse {
#[prost(string, repeated, tag = "1")]
pub lower_dirs: ::prost::alloc::vec::Vec<::prost::alloc::string::String>,
}
/// Whether sandboxes (nested-virtualization microVMs) can run on this host.
#[derive(serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
#[derive(Clone, PartialEq, Eq, Hash, ::prost::Message)]
pub struct SandboxCapability {
/// Whether sandboxes are runnable on the current host and backend.
#[prost(bool, tag = "1")]
pub supported: bool,
/// Actionable reason when unsupported; empty when supported.
#[prost(string, tag = "2")]
pub reason: ::prost::alloc::string::String,
/// The System VM backend the capability was evaluated against.
#[prost(enumeration = "SystemVmBackend", tag = "3")]
pub backend: i32,
}
/// Diagnostic snapshot of the System VM's virtio devices and vCPUs.
#[derive(serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
Expand Down
25 changes: 25 additions & 0 deletions virt/arcbox-hypervisor/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,28 @@ pub fn create_hypervisor() -> Result<impl Hypervisor> {
compile_error!("Unsupported platform: only macOS and Linux are supported")
}
}

/// Reports whether the host supports nested virtualization for guest VMs.
///
/// This gates ArcBox sandboxes: nested Firecracker microVMs need `/dev/kvm`
/// inside the System VM, which the host exposes only when it enables nested
/// virtualization — Apple Silicon M3 or newer on macOS 15+ under the VZ
/// backend. Detection is a cheap static query and does not construct a
/// hypervisor.
#[must_use]
pub fn host_supports_nested_virt() -> bool {
#[cfg(target_os = "macos")]
{
arcbox_vz::GenericPlatform::is_nested_virt_supported()
}

#[cfg(target_os = "linux")]
{
linux::host_supports_nested_virt()
}

#[cfg(not(any(target_os = "macos", target_os = "linux")))]
{
false
}
}
27 changes: 2 additions & 25 deletions virt/arcbox-hypervisor/src/linux/hypervisor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,8 @@ impl KvmHypervisor {
// Determine supported architectures
let supported_archs = vec![CpuArch::native()];

// Check for nested virtualization support
#[cfg(target_arch = "x86_64")]
let nested_virt = Self::check_nested_virt();
#[cfg(not(target_arch = "x86_64"))]
let nested_virt = false;
// Check for nested virtualization support.
let nested_virt = super::host_supports_nested_virt();

Ok(PlatformCapabilities {
supported_archs,
Expand All @@ -112,26 +109,6 @@ impl KvmHypervisor {
})
}

/// Checks if nested virtualization is supported (x86 only).
#[cfg(target_arch = "x86_64")]
fn check_nested_virt() -> bool {
// Check Intel VMX nested support
if let Ok(content) = std::fs::read_to_string("/sys/module/kvm_intel/parameters/nested") {
if content.trim() == "Y" || content.trim() == "1" {
return true;
}
}

// Check AMD SVM nested support
if let Ok(content) = std::fs::read_to_string("/sys/module/kvm_amd/parameters/nested") {
if content.trim() == "Y" || content.trim() == "1" {
return true;
}
}

false
}

/// Returns the KVM system handle.
pub(crate) fn kvm(&self) -> &Arc<KvmSystem> {
&self.kvm
Expand Down
29 changes: 29 additions & 0 deletions virt/arcbox-hypervisor/src/linux/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,32 @@ pub use hypervisor::KvmHypervisor;
pub use memory::KvmMemory;
pub use vcpu::KvmVcpu;
pub use vm::{KvmVm, VirtioDeviceInfo};

/// Whether the Linux host has nested virtualization enabled for KVM.
///
/// Reads the `nested` module parameter of the Intel/AMD KVM drivers. x86-only;
/// other architectures always report `false`.
#[must_use]
pub(crate) fn host_supports_nested_virt() -> bool {
#[cfg(target_arch = "x86_64")]
{
// Intel VMX and AMD SVM expose nesting via a module parameter that
// reads "Y"/"1" when enabled.
for path in [
"/sys/module/kvm_intel/parameters/nested",
"/sys/module/kvm_amd/parameters/nested",
] {
if let Ok(content) = std::fs::read_to_string(path) {
let value = content.trim();
if value == "Y" || value == "1" {
return true;
}
}
}
false
}
#[cfg(not(target_arch = "x86_64"))]
{
false
}
Comment on lines +51 to +54

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 ARM64 Nested KVM Always Rejected

On Linux ARM64, this branch always reports false. A host with nested KVM enabled is therefore exposed as unsupported, so the new CLI pre-check blocks a sandbox that could run and shows an unrelated M3/macOS message.

Artifacts

Repro: rerunnable script that extracts the reviewed function, cross-compiles it, and executes it under ARM64 emulation

  • Contains supporting evidence from the run (text/x-shellscript; charset=utf-8).

Repro: generated repository-derived Rust harness containing the reviewed function and failing capability assertion

  • Contains supporting evidence from the run (text/x-rust; charset=utf-8).

Repro: command transcript showing an aarch64 executable returning false and failing with exit code 101

  • Keeps the command output available without making the summary code-heavy.

View artifacts

T-Rex Ran code and verified through T-Rex

}
Loading