Ticket 8: ash FFI, lifecycle, and FP32 IDENTITY end-to-end (ANV + lavapipe)

[SnowCheetos]

Stacked on #179. Resolves wayfinder ticket 8 (#181) of map #154: the ash FFI, the full Accelerator lifecycle, and FP32 IDENTITY executing end-to-end on a real GPU and a software ICD.

Why

The scaffold (#175) and design ADRs (#177, #179) had no execution behind them. Ticket 8 is the proof that the graph-shaped data plane (ADR 0001), runtime-loaded ash (ADR 0002), checked-in specialized SPIR-V (ADR 0003), the 1.3 baseline and memory-domain map (ADR 0005), and the no-worker-thread fence-poll execution model (ADR 0006) hold on metal.

What

• native.rs — Vulkan 1.3 through pinned ash 0.38.0 (loaded, std): instance/device enumeration with honest DeviceIdentity (device UUID, vendor/device id, GPU class; OTHER for CPU ICDs), compute-queue selection (compute-only family preferred), per-backend device with synchronization2 and bufferDeviceAddress (the latter only to measure allocation alignment). Dedicated VkBuffer+VkDeviceMemory per buffer, bound directly as storage buffers. Host/Shared are persistently mapped host-coherent memory (no flush path needed or issued); Device is device-local memory reached only through a bounded 4 MiB staging copy inside write_buffer/read_buffer. Per-context ring of 64 (command buffer, fence, descriptor set) triples; vkQueueSubmit2 success is the admission boundary; poll_event is one vkGetFenceStatus; finite timeouts rejected pre-admission; VK_ERROR_DEVICE_LOST poisons the instance; parents refuse release while children or in-flight events reference them. Diagnostics: direct_binding_admissions, explicit_transfer_bytes, live_resources.
• lower.rs — hardware-free admission: FP32 target only, one region/block, exactly one boundary-to-boundary IDENTITY, static positive shapes, u32 element domain. VULKAN_TOSA_CAPABILITY names exactly that boundary. The integer target stays declared, unadvertised.
• shader.rs — the checked-in SPIR-V 1.3 word-copy kernel, assembled instruction by instruction with its disassembly beside it (no toolchain, no rspirv), validated structurally by unit tests and offline with naga's SPIR-V front end. Element count is specialization constant 0.
• Finding recorded in ADR 0005: ANV exposes memory types a storage buffer may not use; the heap list alone mis-selects. The memory-domain map is now chosen against a probe buffer's memoryTypeBits.
• SAFETY.md — real audit: ash pin and the exact entry-point inventory, handle lifetime, mappings, ring, device loss, shaders, evidence. Registered in ci/check-release-policy.py (UNSAFE_AUDITS) and docs/release-policy.md (sixth exception). deny.toml allows ISC for libloading.
• CI — vulkan-lavapipe-test lane mirroring openvino-host-test: distribution Mesa, ICD pinned via VK_DRIVER_FILES, VIRTIO_ACCEL_VULKAN=1, and VIRTIO_ACCEL_VULKAN_REQUIRE_DEVICE=1 so a missing ICD fails instead of skipping. Ticket 10 still owns the full lane hardening.
• Docs — README, crate README, docs/portability.md, docs/architecture.md; ADR 0005/0006 gain a ticket 8 evidence section and the review-flagged wording fixes (vkGetPhysicalDeviceFeatures2, the SAFETY.md quote); docs/release-policy.md line break, the crate doc link, and the example's unreachable! from the Ticket 7: scaffold virtio-accel-vulkan (wayfinder map #154) #175 review are fixed.

Validation (2026-09-03, Intel Arc 140V / Lunar Lake, Mesa 26.0.8 ANV, Vulkan 1.4.335; and llvmpipe on the same host)

• cargo test -p virtio-accel-vulkan: 11 unit + 7 host-independent + 12 native tests pass on both devices — lifecycle in every advertised domain (Host, Device, Shared on both), IDENTITY_EDGES_FP32 bit-exact including NaN payload and subnormal, offset bindings with untouched neighbors, segmented staging to device-local memory, binding/timeout rejection, 16 overlapping read-only submissions in flight with Busy on the shared input, ring exhaustion as ResourceLimit, parent-release refusal, and the standard conformance suite with accounting and copy-path hooks in every domain.
• cargo run -p virtio-accel-vulkan --example tosa_vulkan on ANV and pinned to lavapipe: 3.25 round-trips.
• Warm FP32 identity (release, 200 samples): ANV admission p50 16.6 µs / submit-to-complete p50 259 µs; llvmpipe 4.3 µs / 58 µs.
• cargo fmt --check, cargo clippy --workspace --all-targets --all-features -D warnings, python3 ci/check-release-policy.py, cargo deny --workspace --all-features check, cargo test --workspace --all-targets --all-features (one pre-existing OpenVINO failure on this host, executes_boolean_boundaries_on_every_available_device, reproduces on main and is unrelated), placeholder build (VIRTIO_ACCEL_VULKAN=0) clippy/tests/example, cargo doc -D warnings.

Not in this ticket

Operator tiers beyond IDENTITY (ticket 9), FP16/INT8 gating (ticket 5), full CI lane hardening and the numerical corpus (ticket 10), performance doc and manual hardware commands (ticket 11). Grid-stride dispatch for tensors beyond maxComputeWorkGroupCount[0] × 64 elements is rejected as ResourceLimit at load until ticket 9.

Closes #181 once merged.

[Copilot]

🟡 Changes recommended

The Vulkan native test suite includes an environment-variable check that compares an OsString to a &str, which may not compile/behave correctly and should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Implements the first execution-carrying Vulkan backend for the workspace (virtio-accel-vulkan), proving FP32 IDENTITY end-to-end via runtime-loaded ash, checked-in SPIR-V, and the full Accelerator lifecycle on both a real GPU and a Mesa software ICD (lavapipe).

Changes:

• Adds the audited native Vulkan implementation (instance/device bring-up, memory-domain mapping, dedicated buffers, bounded submission ring, fence-poll completion, device-loss poisoning).
• Introduces a checked-in SPIR-V 1.3 word-copy kernel and hardware-free TOSA admission for the FP32 IDENTITY boundary, plus extensive native + host-independent tests.
• Extends CI and docs/release policy plumbing to cover the new host-native unsafe exception and validate against lavapipe.

File summaries

File	Description
README.md	Updates top-level project messaging to reflect Vulkan FP32 IDENTITY end-to-end support.
docs/release-policy.md	Adds Vulkan as a sixth reviewed unsafe exception and minor formatting fix.
docs/portability.md	Updates Vulkan portability description from scaffold to executing backend + domains.
docs/architecture.md	Documents Vulkan provider architecture and how it fits the seam/ADRs.
docs/adr/0006-execution-event-model.md	Records ticket-8 evidence for fence-poll completion model.
docs/adr/0005-vulkan-baseline-probe.md	Adds ticket-8 evidence and clarifies feature/property queries.
deny.toml	Allows ISC license for libloading (pulled in via ash).
crates/virtio-accel-vulkan/tests/vulkan.rs	Replaces scaffold tests with full native integration/conformance suite.
crates/virtio-accel-vulkan/tests/targets.rs	Adds host-independent tests for targets/capability and SPIR-V stability.
crates/virtio-accel-vulkan/src/shader.rs	Adds checked-in SPIR-V assembler + unit tests for the copy kernel.
crates/virtio-accel-vulkan/src/native.rs	Adds the audited Vulkan Accelerator implementation and lifecycle.
crates/virtio-accel-vulkan/src/lower.rs	Implements the FP32 IDENTITY-only admission boundary and lowering plan.
crates/virtio-accel-vulkan/src/lib.rs	Switches from scaffold-only to native+placeholder split and exports.
crates/virtio-accel-vulkan/SAFETY.md	Replaces scaffold audit stub with the full unsafe-code audit.
crates/virtio-accel-vulkan/README.md	Updates crate README to describe what executes today and how to run it.
crates/virtio-accel-vulkan/examples/tosa_vulkan.rs	Updates example to run FP32 IDENTITY (skip gracefully if unavailable).
crates/virtio-accel-vulkan/Cargo.toml	Adds pinned ash dependency behind OS cfg gate.
ci/check-release-policy.py	Registers Vulkan unsafe exception and audit anchors.
Cargo.lock	Adds ash/libloading and transitive deps.
.github/workflows/ci.yml	Adds a lavapipe CI lane to ensure native Vulkan path is exercised.

Review details

• Files reviewed: 19/21 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[aravishankar-mp]

Tested this on AMD hardware, since the validation in the description covers Intel ANV and lavapipe.

Host: Ryzen AI 7 350 (Krackan) / Radeon 860M iGPU, amdgpu, RADV, Mesa 26.1.8, Vulkan 1.4.354, kernel 7.1.12. Tested at 040c19c.

What passes

Ran the vulkan-lavapipe-test recipe verbatim with VK_DRIVER_FILES repointed at RADV:

• 30/30 tests pass (11 unit + 7 target + 12 native) pinned to RADV, pinned to lavapipe, and unpinned across both.
• Device identity is honest: vendor 0x1002, device 0x1114, AcceleratorClass(2).
• All three memory domains advertise (HOST_VISIBLE | DEVICE_LOCAL | SHARED), and no "no device-local memory type" diagnostic fires — so the bounded staging path is genuinely exercised despite this being a unified-memory APU. The probe-buffer memoryTypeBits selection from the ANV finding holds on RADV.
• tosa_vulkan round-trips 3.25 on the 860M.
• The Vulkan crate is clippy-clean under Rust 1.96.

One flaky test found, commented inline — it is a test-side race, not a backend defect, and it reproduces on lavapipe too.

Unrelated to this PR: cargo clippy -D warnings trips nonminimal_bool at crates/virtio-accel-tosa/src/semantic.rs:1385. That line is identical on main and this PR doesn't touch the tosa crate; it's a newer-lint artifact of local Rust 1.96 vs CI's stable.

Performance

Warm FP32 identity, release, 200 samples:

	admission p50	submit-to-complete p50
RADV / Radeon 860M	~4.0 µs	~15.5 µs
llvmpipe (same host)	~0.9 µs	~19 µs

These are not directly comparable to the ANV numbers in the description — different machine, CPU, kernel, and Mesa (26.0.8 vs 26.1.8). llvmpipe runs on both hosts and touches no GPU driver, so it works as a host baseline. Normalizing each device against its own host's llvmpipe:

• Admission: RADV is ~4.4× its host baseline; ANV is ~3.9× its own. Effectively identical — the raw 16.6 µs → 4.0 µs difference is host CPU speed, not the backend.
• Completion: RADV lands at 0.82× its baseline; ANV at 4.5×. That ~5× normalized gap is real, and host speed does not explain it.

Instrumenting the poll loop on RADV: each vkGetFenceStatus costs 283 ns (a userspace read of a mapped seqno, no ioctl), ~30 polls to observe completion, with the poll calls accounting for 51% of the measured window and the remainder being yield_now() scheduling. A 259 µs completion on a 4-byte dispatch is far too long to be compute; it is consistent with fence signalling that only becomes visible after an interrupt is processed, and/or a vkGetFenceStatus that traps into a DRM ioctl per poll. I can't confirm which — no Lunar Lake hardware here, and I did not reproduce the ANV numbers, only read them from the description.

Caveats worth carrying into ticket 11 if any of this lands in a perf doc:

• This is a dispatch-overhead benchmark, not throughput. A 4-byte one-element identity measures submit + fence round-trip and nothing else. The Arc 140V is a substantially larger GPU than the 860M and would very likely win on real compute; nothing here suggests otherwise.
• The metric embeds the harness's polling strategy. Busy-poll plus yield_now() means "completion" is partly a scheduler measurement; a blocking vkWaitForFences would report differently on both devices.
• RADV showed p99 excursions to 81–162 µs, consistent with GPU power-state transitions on power_dpm_force_performance_level=auto.

[aravishankar-mp]

Flaky on fast devices — 23–33% failure rate on AMD (RADV, Radeon 860M).

Tested on a Ryzen AI 7 350 (Krackan) / Radeon 860M, Mesa 26.1.8, RADV. This test fails intermittently with:

AMD Radeon 860M Graphics (RADV KRACKAN1): unloaded a program with events in flight

7/30 failures in one 30-run loop, 10/30 in another.

The backend is behaving correctly; the test's assumption is what breaks. in_flight is decremented in VulkanEvent::latch() (native.rs), and latch() only ever runs from poll(). So a submission counts as in flight until its terminal state is observed, not until the GPU finishes it.

This block polls all IN_FLIGHT events first. On a device quick enough to retire all 16 before the poll runs, every guard drops, in_flight reaches 0, and unload_program just below legitimately returns Ok(()) — firing the Ok(()) => panic!(...) arm. Worth noting the write_buffer assertion in this block is already guarded against exactly this case; the unload_program arm below it is not.

Not AMD-specific. Injecting a 50 ms sleep before this block makes it fail 100% on lavapipe, the ICD the vulkan-lavapipe-test lane uses — so it is latent in CI rather than immune to it. AMD just retires 16 submissions (~250 µs) fast enough to lose the race routinely.

Suggested fix: assert both refusals before anything is polled. That is deterministic on every device, since un-polled events always hold their guards.

Suggested change

	// While the reads are in flight the shared input may not be transferred or freed, and
	// the program may not be unloaded.
	if events
	.iter()
	.any(|event| backend.poll_event(event).unwrap() == EventState::Pending)
	{
	assert_eq!(
	backend
	.write_buffer(&mut shared_input, 0, &SliceSource(&payload))
	.unwrap_err(),
	BackendError::Busy
	);
	}
	// While the reads are in flight the shared input may not be transferred or freed, and
	// the program may not be unloaded. Both refusals are asserted before anything is polled:
	// a submission stays in flight only until its terminal state is *observed*, so polling
	// first lets a device quick enough to retire all `IN_FLIGHT` submissions drop the guards
	// and make either release legitimately succeed.
	assert_eq!(
	backend
	.write_buffer(&mut shared_input, 0, &SliceSource(&payload))
	.unwrap_err(),
	BackendError::Busy,
	"{device}"
	);

Validated: 40/40 on RADV, and passes on both RADV and lavapipe even with the 50 ms delay injected. Full native suite 25/25 on AMD, cargo fmt --check clean, no new warnings.

[aravishankar-mp]

This rewrite makes the Decision section contradict the Context bullet it was meant to fix.

Float controls are properties, not features — they live in VkPhysicalDeviceVulkan12Properties and are read via vkGetPhysicalDeviceProperties2, exactly as the new Context bullet above now states. Attributing them to vkGetPhysicalDeviceFeatures2 here re-introduces the same error a few lines below the correction. The pre-rebase wording had this right.

Suggest keeping the split explicit:

Suggested change

	when `shaderFloat16` and `vkGetPhysicalDeviceFeatures2` float-controls prove per-device (ADR 0004);
	when `shaderFloat16` (`vkGetPhysicalDeviceFeatures2`) and the `VkPhysicalDeviceVulkan12Properties`
	float-controls fields (`vkGetPhysicalDeviceProperties2`) prove per-device (ADR 0004);

(For what it's worth, PhysicalDeviceRecord::probe currently chains PhysicalDeviceVulkan11Properties onto get_physical_device_properties2 and Vulkan12Features/Vulkan13Features onto get_physical_device_features2 — float controls aren't probed at all yet, which is consistent with this being ticket 5's job.)