diff --git a/docs/planning/xhci-head-to-head.md b/docs/planning/xhci-head-to-head.md new file mode 100644 index 00000000..dc083e0b --- /dev/null +++ b/docs/planning/xhci-head-to-head.md @@ -0,0 +1,105 @@ +# xHCI Head-to-Head: Linux Module vs Breenix Driver + +## Status Summary + +| | Linux Module | Breenix Driver | +|---|---|---| +| **Location** | `linux_xhci_module/breenix_xhci_probe.c` | `kernel/src/drivers/usb/xhci.rs` | +| **Platform** | linux-probe VM (Alpine ARM64) | breenix-dev VM (Breenix ARM64) | +| **M1-M10** | All PASS | All PASS | +| **M11 (Event Delivery)** | PASS — HID reports received via MSI | FAIL — CC=12, all endpoints Halted | +| **Proven 100%?** | Yes, with and without xhci_hcd priming | No | + +## Linux Module: Proven Working + +The Linux module has been validated in two configurations: + +1. **With xhci_hcd priming** — stock Linux xhci_hcd loads first, our module replaces it +2. **Without xhci_hcd priming** — stock xhci_hcd unbound before our module loads + +Both produce identical results: all 11 milestones pass, HID interrupt events arrive via MSI handler, keyboard/mouse reports are printed to dmesg. + +## Instrumentation Parity Gaps + +The Linux module dumps full device contexts (192-256 byte hex dumps) at every configuration step. The Breenix driver only logs completion codes. **This is the core diagnostic gap.** + +### Critical Missing Dumps in Breenix + +| Milestone | Linux Dumps | Breenix Dumps | Gap | +|---|---|---|---| +| M2 (Reset) | USBSTS + all registers (`ms_regs`) | USBSTS only | Missing register snapshot | +| M3 (Data Structures) | Ring/ERST + all registers | Ring/ERST only | Missing register snapshot | +| M4 (Running) | USBCMD/STS/IMAN + all registers | USBCMD/STS/IMAN only | Missing register snapshot | +| **M7 (Address Device)** | **Input ctx (192B) + cmd TRB + evt TRB + output ctx (192B)** | **CC only** | **MAJOR — no context visibility** | +| **M8 (Endpoint Config)** | **Input ctx (256B) + output ctx (256B) + BW dance contexts** | **CC only** | **MAJOR — no endpoint context visibility** | +| **M9 (HID Setup)** | **Output ctx after SET_CONFIG + after HID setup** | **CC only** | **MAJOR — no post-config state** | +| M10 (Interrupt Transfer) | TRB + EP state + registers | TRB + pre/post EP state + registers | None (Breenix more detailed) | +| M11 (Event Delivery) | Register snapshot + EP contexts + DCBAA | Same + pending event check | None | + +### What This Means + +We cannot currently do a byte-for-byte comparison of device context contents between the two platforms. The Linux module shows exactly what input context was sent and what output context the controller returned. Breenix only shows "CC=1" (success) — we can't see if the actual context data differs. + +## What We Know Works Identically + +- Controller discovery (BAR, capabilities, version) +- Controller reset (HCRST completes, CNR clears) +- Data structure setup (DCBAA, command ring, event ring, ERST) +- Controller start (RS=1, INTE=1, IMAN.IE=1) +- Port detection (CCS=1, PED=1, speed correct) +- Slot enablement (EnableSlot CC=1) +- Device addressing (AddressDevice CC=1) +- Endpoint configuration (ConfigureEndpoint CC=1, BW dance CC=1) +- HID class setup (SET_CONFIGURATION, SET_IDLE, SET_PROTOCOL all succeed) +- Interrupt TRB queueing (Normal TRBs enqueued, doorbells rung) + +## What Fails + +- **M11 only**: First interrupt transfer event returns CC=12 (Endpoint Not Enabled) +- All 4 interrupt endpoints transition immediately from Running to Halted +- Continuous polling confirms CC=12 never clears + +## Eliminated Hypotheses (26 total) + +1-18: Prior session hypotheses (xHCI logic, DMA, cache, register ordering, etc.) +19. Set TR Dequeue Pointer explicit command +20. USB device state (SET_CONFIGURATION(0) before ConfigureEndpoint) +21. xhci_hcd priming (Linux module works without it) +22. PCI configuration differences (identical Command register) +23. MSI configuration/ordering +24. Timing (10s, 60s delays) +25. phymemrange_enable alone (fires but no ep create) +26. EHCI companion init (CONFIGFLAG=1, RS=1, HCRST — no effect) + +## Next Steps: Close the Instrumentation Gap + +### Priority 1: Add Context Dumps to Breenix + +Add `ms_dump` equivalent for device contexts at M7, M8, M9: + +- **M7**: Dump input context before AddressDevice, output context after +- **M8**: Dump input context before ConfigureEndpoint, output context after, plus BW dance contexts +- **M9**: Dump output context after SET_CONFIGURATION and after HID setup + +### Priority 2: Extract Matching Data from Linux Module + +Run the Linux module on linux-probe and capture the full dmesg output with all context dumps. This becomes the **reference dataset**. + +### Priority 3: Byte-for-Byte Comparison + +Compare every dumped context field between Linux and Breenix: +- Slot Context: Route String, Speed, Context Entries, Max Exit Latency +- Endpoint 0 Context: EP Type, Max Packet Size, TR Dequeue, Interval, etc. +- Interrupt EP Contexts: EP Type, Max Packet Size, Interval, Mult, MaxPStreams, etc. + +Any difference is a candidate root cause for CC=12. + +### Priority 4: Stop Using serial_println for Debug + +All experimental debug output has been added via `serial_println!`. This violates the project's tracing policy. The Parallels workaround code should use the lock-free tracing subsystem instead. + +## Parallels Host Log Observations + +The Parallels host log shows the hypervisor has an internal "ep create" mechanism. On linux-probe, `ep create` events fire ~330ms after xHCI init. On breenix-dev, they never fire. This is a symptom, not a root cause — the hypervisor creates endpoints when the xHCI controller state is correct, and doesn't when it's not. + +Rather than reverse-engineering Parallels' internal signaling, the right approach is to make Breenix's xHCI state **byte-identical** to Linux's. The context dumps will show us where they differ. diff --git a/kernel/src/arch_impl/aarch64/boot.S b/kernel/src/arch_impl/aarch64/boot.S index d3a2e128..d6d2068f 100644 --- a/kernel/src/arch_impl/aarch64/boot.S +++ b/kernel/src/arch_impl/aarch64/boot.S @@ -35,10 +35,19 @@ .equ BLOCK_FLAGS_DEVICE, (DESC_BLOCK | DESC_AF | DESC_SH_NONE | DESC_AP_KERNEL | DESC_ATTR_DEVICE | DESC_PXN | DESC_UXN) .equ BLOCK_FLAGS_NORMAL, (DESC_BLOCK | DESC_AF | DESC_SH_INNER | DESC_AP_KERNEL | DESC_ATTR_NORMAL) +// Non-Cacheable block flags for DMA buffers (MAIR index 2 = 0x44) +.equ DESC_ATTR_NC, (2 << 2) +.equ BLOCK_FLAGS_NC, (DESC_BLOCK | DESC_AF | DESC_SH_INNER | DESC_AP_KERNEL | DESC_ATTR_NC) + +// DMA NC region: 2MB block at physical 0x5000_0000 +// L2 index = (0x50000000 - 0x40000000) / 0x200000 = 128 +.equ NC_L2_INDEX, 128 + // MAIR attributes .equ MAIR_ATTR_DEVICE, 0x00 .equ MAIR_ATTR_NORMAL, 0xFF -.equ MAIR_EL1_VALUE, (MAIR_ATTR_DEVICE | (MAIR_ATTR_NORMAL << 8)) +.equ MAIR_ATTR_NC, 0x44 +.equ MAIR_EL1_VALUE, (MAIR_ATTR_DEVICE | (MAIR_ATTR_NORMAL << 8) | (MAIR_ATTR_NC << 16)) // TCR configuration (4KB granule, 48-bit VA, inner-shareable, WBWA) .equ TCR_T0SZ, 16 @@ -262,10 +271,14 @@ setup_mmu: bl zero_table ldr x0, =ttbr0_l1 bl zero_table + ldr x0, =ttbr0_l2_ram + bl zero_table ldr x0, =ttbr1_l0 bl zero_table ldr x0, =ttbr1_l1 bl zero_table + ldr x0, =ttbr1_l2_ram + bl zero_table // TTBR0 L0[0] -> L1 ldr x0, =ttbr0_l0 @@ -280,12 +293,16 @@ setup_mmu: orr x1, x1, x2 str x1, [x0, #0] - // TTBR0 L1[1] = normal (0x4000_0000 - 0x7FFF_FFFF) - ldr x1, =0x40000000 - ldr x2, =BLOCK_FLAGS_NORMAL - orr x1, x1, x2 + // TTBR0 L1[1] -> L2 table (0x4000_0000 - 0x7FFF_FFFF) + // Uses L2 to carve out a 2MB NC block for DMA at 0x5000_0000 + ldr x1, =ttbr0_l2_ram + orr x1, x1, #DESC_TABLE str x1, [x0, #8] + // Fill TTBR0 L2: 512 x 2MB blocks, index NC_L2_INDEX = NC, rest = Normal + ldr x0, =ttbr0_l2_ram + bl fill_l2_ram + // TTBR1 L0[0] -> L1 ldr x0, =ttbr1_l0 ldr x1, =ttbr1_l1 @@ -299,13 +316,18 @@ setup_mmu: orr x1, x1, x2 str x1, [x0, #0] - // TTBR1 L1[1] = normal (high-half direct map) - ldr x1, =0x40000000 - ldr x2, =BLOCK_FLAGS_NORMAL - orr x1, x1, x2 + // TTBR1 L1[1] -> L2 table (high-half direct map 0x4000_0000 - 0x7FFF_FFFF) + // Same NC carveout for .dma section at 0xFFFF_0000_5000_0000 + ldr x1, =ttbr1_l2_ram + orr x1, x1, #DESC_TABLE str x1, [x0, #8] + // Fill TTBR1 L2: same layout as TTBR0 + ldr x0, =ttbr1_l2_ram + bl fill_l2_ram + // TTBR1 L1[2] = normal (high-half direct map) + ldr x0, =ttbr1_l1 ldr x1, =0x80000000 ldr x2, =BLOCK_FLAGS_NORMAL orr x1, x1, x2 @@ -355,6 +377,26 @@ zero_table_loop: b.ne zero_table_loop ret +// Fill L2 table at x0 with 512 x 2MB block entries for 0x4000_0000 - 0x7FFF_FFFF. +// Entry NC_L2_INDEX (index 128 = physical 0x5000_0000) gets BLOCK_FLAGS_NC. +// All other entries get BLOCK_FLAGS_NORMAL. +fill_l2_ram: + mov x1, #0 // i = 0 + ldr x3, =0x40000000 // base physical address + ldr x4, =BLOCK_FLAGS_NORMAL + ldr x5, =BLOCK_FLAGS_NC +fill_l2_loop: + lsl x6, x1, #21 // offset = i * 2MB (0x200000) + add x6, x6, x3 // phys = 0x40000000 + offset + cmp x1, #NC_L2_INDEX + csel x7, x5, x4, eq // flags = NC if i==128, else Normal + orr x6, x6, x7 // entry = phys | flags + str x6, [x0, x1, lsl #3] // L2[i] = entry + add x1, x1, #1 + cmp x1, #512 + b.lt fill_l2_loop + ret + .section .text /* @@ -844,12 +886,18 @@ ttbr0_l0: .global ttbr0_l1 ttbr0_l1: .skip 4096 +.global ttbr0_l2_ram +ttbr0_l2_ram: + .skip 4096 .global ttbr1_l0 ttbr1_l0: .skip 4096 .global ttbr1_l1 ttbr1_l1: .skip 4096 +.global ttbr1_l2_ram +ttbr1_l2_ram: + .skip 4096 .balign 64 .global CPU_RELEASE_ADDR diff --git a/kernel/src/arch_impl/aarch64/linker.ld b/kernel/src/arch_impl/aarch64/linker.ld index 445103a8..66095d6e 100644 --- a/kernel/src/arch_impl/aarch64/linker.ld +++ b/kernel/src/arch_impl/aarch64/linker.ld @@ -88,6 +88,17 @@ SECTIONS __stack_top = .; } + /* DMA buffers - placed in Non-Cacheable region at physical 0x50000000. + * The loader maps this 2MB block with MAIR index 2 (NC) so the xHC + * controller and CPU see consistent data without cache maintenance. + * Zeroed at runtime by kernel init, not by loader (NOLOAD section). */ + . = KERNEL_VIRT_BASE + 0x50000000; + .dma (NOLOAD) : AT(0x50000000) { + __dma_start = .; + *(.dma .dma.*) + __dma_end = .; + } + /DISCARD/ : { *(.comment) *(.note*) diff --git a/kernel/src/arch_impl/aarch64/mmu.rs b/kernel/src/arch_impl/aarch64/mmu.rs index 85c8f261..3a18b896 100644 --- a/kernel/src/arch_impl/aarch64/mmu.rs +++ b/kernel/src/arch_impl/aarch64/mmu.rs @@ -34,7 +34,8 @@ static mut L2_TABLE_RAM: PageTable = PageTable::new(); const MAIR_ATTR_DEVICE: u64 = 0x00; const MAIR_ATTR_NORMAL: u64 = 0xFF; -const MAIR_EL1_VALUE: u64 = MAIR_ATTR_DEVICE | (MAIR_ATTR_NORMAL << 8); +const MAIR_ATTR_NC: u64 = 0x44; +const MAIR_EL1_VALUE: u64 = MAIR_ATTR_DEVICE | (MAIR_ATTR_NORMAL << 8) | (MAIR_ATTR_NC << 16); const TCR_T0SZ: u64 = 16; const TCR_T1SZ: u64 = 16 << 16; diff --git a/kernel/src/drivers/mod.rs b/kernel/src/drivers/mod.rs index 250d5fc8..e80783fd 100644 --- a/kernel/src/drivers/mod.rs +++ b/kernel/src/drivers/mod.rs @@ -92,50 +92,18 @@ pub fn init() -> usize { let device_count = pci::enumerate(); serial_println!("[drivers] Found {} PCI devices", device_count); - // Log all PCI devices for debugging - if let Some(devices) = pci::get_devices() { - for dev in &devices { - serial_println!( - "[drivers] PCI {:02x}:{:02x}.{} [{:04x}:{:04x}] class={:?}/0x{:02x}", - dev.bus, dev.device, dev.function, - dev.vendor_id, dev.device_id, - dev.class, dev.subclass, - ); - } - } - // Enumerate VirtIO PCI devices with modern transport let virtio_devices = virtio::pci_transport::enumerate_virtio_pci_devices(); - for dev in &virtio_devices { - serial_println!( - "[drivers] VirtIO PCI device: {} (type={})", - virtio::pci_transport::device_type_name(dev.device_id()), - dev.device_id() - ); - } serial_println!("[drivers] Found {} VirtIO PCI devices", virtio_devices.len()); - // Initialize VirtIO GPU PCI driver. - // Even when a GOP framebuffer is available (Parallels), we try the VirtIO - // GPU PCI driver first — it supports arbitrary resolutions via - // CREATE_RESOURCE_2D, giving us control beyond the fixed GOP mode. - // If GPU PCI init fails, the GOP framebuffer is used as a fallback. match virtio::gpu_pci::init() { - Ok(()) => { - serial_println!("[drivers] VirtIO GPU (PCI) initialized"); - } - Err(e) => { - serial_println!("[drivers] VirtIO GPU (PCI) init failed: {}", e); - } + Ok(()) => serial_println!("[drivers] VirtIO GPU (PCI) initialized"), + Err(e) => serial_println!("[drivers] VirtIO GPU (PCI) init failed: {}", e), } - // EHCI USB 2.0 host controller SKIPPED — investigating whether EHCI - // init causes a second xHC reset in the Parallels hypervisor, which - // would destroy interrupt endpoint configurations and cause CC=12. - // Intel 82801FB EHCI: vendor 0x8086, device 0x265c - if pci::find_device(0x8086, 0x265c).is_some() { - serial_println!("[drivers] EHCI USB 2.0 controller found but SKIPPED (CC=12 investigation)"); - } + // EHCI USB 2.0 controller — initialization is handled inside xhci::init() + // as a prerequisite for Parallels USB device routing (companion controller model). + // Intel 82801FB EHCI: vendor 0x8086, device 0x265c at PCI 00:02.0 // Initialize XHCI USB host controller (keyboard + mouse) // NEC uPD720200: vendor 0x1033, device 0x0194 diff --git a/kernel/src/drivers/pci.rs b/kernel/src/drivers/pci.rs index 6366ffbd..62dfdaee 100644 --- a/kernel/src/drivers/pci.rs +++ b/kernel/src/drivers/pci.rs @@ -344,6 +344,134 @@ impl Device { let new_ctrl = (msg_ctrl & !0x0070) | 0x0001; // Clear MME, set Enable pci_write_config_word(self.bus, self.device, self.function, cap_offset + 2, new_ctrl); } + + /// Find any PCI capability by ID. Returns the config space offset, or None. + pub fn find_capability(&self, cap_id: u8) -> Option { + let status = pci_read_config_word(self.bus, self.device, self.function, 0x06); + if (status & (1 << 4)) == 0 { + return None; + } + let mut cap_ptr = pci_read_config_byte(self.bus, self.device, self.function, 0x34); + while cap_ptr != 0 { + let id = pci_read_config_byte(self.bus, self.device, self.function, cap_ptr); + if id == cap_id { + return Some(cap_ptr); + } + cap_ptr = pci_read_config_byte(self.bus, self.device, self.function, cap_ptr + 1); + } + None + } + + /// Transition device to D0 power state via PM capability (cap ID 0x01). + /// This is what Linux's pci_enable_device() does internally via pci_set_power_state(). + /// Returns the previous power state (0=D0, 1=D1, 2=D2, 3=D3hot), or None if no PM cap. + pub fn set_power_state_d0(&self) -> Option { + let pm_cap = self.find_capability(0x01)?; // PCI_CAP_ID_PM = 0x01 + // PMCSR (Power Management Control/Status Register) is at PM_cap + 4 + let pmcsr = pci_read_config_word(self.bus, self.device, self.function, pm_cap + 4); + let current_state = (pmcsr & 0x03) as u8; // Bits [1:0] = power state + if current_state != 0 { + // Not in D0 — transition to D0 by clearing bits [1:0] + let new_pmcsr = pmcsr & !0x03; + pci_write_config_word(self.bus, self.device, self.function, pm_cap + 4, new_pmcsr); + // PCI spec requires 10ms delay after D3hot->D0 transition + // (We always wait this since it's safe) + for _ in 0..10_000_000u64 { + core::hint::spin_loop(); + } + } + Some(current_state) + } + + /// Set Cache Line Size register (offset 0x0C). + /// Linux sets this based on the CPU's cache line size (typically 64 bytes = 16 DWORDs). + pub fn set_cache_line_size(&self, size_dwords: u8) { + pci_write_config_byte(self.bus, self.device, self.function, 0x0C, size_dwords); + } + + /// Set Latency Timer register (offset 0x0D). + /// Linux's pci_set_master() sets this to 64 on conventional PCI if it's < 16. + pub fn set_latency_timer(&self, timer: u8) { + pci_write_config_byte(self.bus, self.device, self.function, 0x0D, timer); + } + + /// Dump all PCI capabilities for diagnostics (prints to serial). + pub fn dump_capabilities(&self) { + let status = pci_read_config_word(self.bus, self.device, self.function, 0x06); + if (status & (1 << 4)) == 0 { + crate::serial_println!("[pci] {:02x}:{:02x}.{}: no capabilities list", self.bus, self.device, self.function); + return; + } + let mut cap_ptr = pci_read_config_byte(self.bus, self.device, self.function, 0x34); + crate::serial_println!("[pci] {:02x}:{:02x}.{}: capabilities:", self.bus, self.device, self.function); + while cap_ptr != 0 { + let id = pci_read_config_byte(self.bus, self.device, self.function, cap_ptr); + let cap_name = match id { + 0x01 => "PM", + 0x05 => "MSI", + 0x10 => "PCIe", + 0x11 => "MSI-X", + 0x12 => "SATA", + _ => "?", + }; + // Read the full dword at cap_ptr for extra context + let dw0 = pci_read_config_dword(self.bus, self.device, self.function, cap_ptr); + let dw1 = pci_read_config_dword(self.bus, self.device, self.function, cap_ptr + 4); + crate::serial_println!(" cap 0x{:02x} ({}) @ 0x{:02x}: dw0=0x{:08x} dw1=0x{:08x}", + id, cap_name, cap_ptr, dw0, dw1); + cap_ptr = pci_read_config_byte(self.bus, self.device, self.function, cap_ptr + 1); + } + } + + /// Dump full 256-byte PCI config space in milestone format for byte-for-byte comparison. + /// Output format: `[M1] label +offset: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX` + pub fn dump_config_space_256(&self, label: &str) { + for offset in (0u8..=240).step_by(16) { + let dw0 = pci_read_config_dword(self.bus, self.device, self.function, offset); + let dw1 = pci_read_config_dword(self.bus, self.device, self.function, offset + 4); + let dw2 = pci_read_config_dword(self.bus, self.device, self.function, offset + 8); + let dw3 = pci_read_config_dword(self.bus, self.device, self.function, offset + 12); + crate::serial_println!("[M1] {} +{:03x}: {:08x} {:08x} {:08x} {:08x}", + label, offset, dw0, dw1, dw2, dw3); + } + } + + /// Full Linux-style PCI device enable: D0 transition + bus master + memory space + INTx disable. + /// This replicates what Linux's pci_enable_device() + pci_set_master() does. + pub fn linux_style_enable(&self) { + // 1. Transition to D0 power state (like pci_set_power_state(dev, PCI_D0)) + if let Some(prev_state) = self.set_power_state_d0() { + crate::serial_println!("[pci] {:02x}:{:02x}.{}: PM D{} -> D0", + self.bus, self.device, self.function, prev_state); + } else { + crate::serial_println!("[pci] {:02x}:{:02x}.{}: no PM capability", + self.bus, self.device, self.function); + } + + // 2. Set Cache Line Size (64 bytes = 16 DWORDs, standard for ARM64) + self.set_cache_line_size(16); + + // 3. Set Latency Timer (Linux uses 64 for conventional PCI) + self.set_latency_timer(64); + + // 4. Enable Memory Space + Bus Master + Disable INTx (all in one write) + let command = pci_read_config_word(self.bus, self.device, self.function, 0x04); + // Bit 1: Memory Space, Bit 2: Bus Master, Bit 10: INTx Disable + let new_command = command | 0x0406; + pci_write_config_word(self.bus, self.device, self.function, 0x04, new_command); + crate::serial_println!("[pci] {:02x}:{:02x}.{}: cmd 0x{:04x} -> 0x{:04x}", + self.bus, self.device, self.function, command, new_command); + + // 5. Clear any error bits in Status register (write-1-to-clear) + let status = pci_read_config_word(self.bus, self.device, self.function, 0x06); + if status & 0xF900 != 0 { + // Clear error bits: SERR (14), Parity (15), Sig Target Abort (11), + // Rcvd Target Abort (12), Rcvd Master Abort (13), Sig System Error (14), Parity (15) + pci_write_config_word(self.bus, self.device, self.function, 0x06, status); + crate::serial_println!("[pci] {:02x}:{:02x}.{}: cleared status errors 0x{:04x}", + self.bus, self.device, self.function, status & 0xF900); + } + } } impl fmt::Display for Device { @@ -408,10 +536,10 @@ pub(crate) fn pci_read_config_dword(bus: u8, device: u8, function: u8, offset: u } let addr = ecam_base - + ((bus as u64) << 20) + + (((bus as u64) << 20) | ((device as u64) << 15) | ((function as u64) << 12) - | ((offset & 0xFC) as u64); + | ((offset & 0xFC) as u64)); const HHDM_BASE: u64 = 0xFFFF_0000_0000_0000; let virt = (HHDM_BASE + addr) as *const u32; @@ -445,10 +573,10 @@ pub(crate) fn pci_write_config_dword(bus: u8, device: u8, function: u8, offset: } let addr = ecam_base - + ((bus as u64) << 20) + + (((bus as u64) << 20) | ((device as u64) << 15) | ((function as u64) << 12) - | ((offset & 0xFC) as u64); + | ((offset & 0xFC) as u64)); const HHDM_BASE: u64 = 0xFFFF_0000_0000_0000; let virt = (HHDM_BASE + addr) as *mut u32; @@ -482,6 +610,45 @@ pub(crate) fn pci_read_config_byte(bus: u8, device: u8, function: u8, offset: u8 ((dword >> shift) & 0xFF) as u8 } +/// Write an 8-bit value to PCI configuration space +#[allow(dead_code)] // Part of low-level API +pub(crate) fn pci_write_config_byte(bus: u8, device: u8, function: u8, offset: u8, value: u8) { + let dword_offset = offset & 0xFC; + let mut dword = pci_read_config_dword(bus, device, function, dword_offset); + let shift = ((offset & 3) * 8) as u32; + let mask = !(0xFF << shift); + dword = (dword & mask) | ((value as u32) << shift); + pci_write_config_dword(bus, device, function, dword_offset, dword); +} + +/// Read a BAR address without writing 0xFFFFFFFF for sizing. +/// Used for devices where BAR sizing disrupts the device's internal state +/// (e.g., Parallels vxHC). Sets size to 0x1000 (4KB minimum) since we +/// can't determine the actual size without the destructive write. +fn read_bar_no_sizing(bus: u8, device: u8, function: u8, bar_index: u8) -> (Bar, bool) { + let offset = 0x10 + (bar_index * 4); + let bar_low = pci_read_config_dword(bus, device, function, offset); + + if bar_low & 0x01 != 0 { + // I/O space BAR + let address = (bar_low & 0xFFFF_FFFC) as u64; + (Bar { address, size: 0x100, is_io: true, is_64bit: false, prefetchable: false }, false) + } else { + let bar_type = (bar_low >> 1) & 0x03; + let prefetchable = (bar_low & 0x08) != 0; + if bar_type == 0x02 { + // 64-bit BAR + let bar_high = pci_read_config_dword(bus, device, function, offset + 4); + let address = ((bar_high as u64) << 32) | ((bar_low & 0xFFFF_FFF0) as u64); + (Bar { address, size: 0x1000, is_io: false, is_64bit: true, prefetchable }, true) + } else { + // 32-bit BAR + let address = (bar_low & 0xFFFF_FFF0) as u64; + (Bar { address, size: 0x1000, is_io: false, is_64bit: false, prefetchable }, false) + } + } +} + /// Decode a BAR from PCI configuration space fn decode_bar(bus: u8, device: u8, function: u8, bar_index: u8) -> (Bar, bool) { let offset = 0x10 + (bar_index * 4); @@ -610,14 +777,27 @@ fn probe_device(bus: u8, device: u8, function: u8) -> Option { let interrupt_pin = (int_reg >> 8) as u8; // Decode BARs + // Skip destructive BAR sizing (write 0xFFFFFFFF) for USB controllers + // (class=0x0C, subclass=0x03). On Parallels, the BAR disable/re-enable + // from sizing corrupts the vxHC's internal USB emulation state. + let skip_xhci_sizing = class_code == 0x0C && subclass == 0x03; let mut bars = [Bar::empty(); 6]; let mut bar_index = 0; while bar_index < 6 { - let (bar, skip_next) = decode_bar(bus, device, function, bar_index); - bars[bar_index as usize] = bar; - bar_index += 1; - if skip_next && bar_index < 6 { - bar_index += 1; // Skip the next BAR slot for 64-bit BARs + if skip_xhci_sizing { + let (bar, skip_next) = read_bar_no_sizing(bus, device, function, bar_index); + bars[bar_index as usize] = bar; + bar_index += 1; + if skip_next && bar_index < 6 { + bar_index += 1; + } + } else { + let (bar, skip_next) = decode_bar(bus, device, function, bar_index); + bars[bar_index as usize] = bar; + bar_index += 1; + if skip_next && bar_index < 6 { + bar_index += 1; // Skip the next BAR slot for 64-bit BARs + } } } diff --git a/kernel/src/drivers/usb/xhci.rs b/kernel/src/drivers/usb/xhci.rs index c8c92b81..ebb33941 100644 --- a/kernel/src/drivers/usb/xhci.rs +++ b/kernel/src/drivers/usb/xhci.rs @@ -48,6 +48,12 @@ const HHDM_BASE: u64 = 0xFFFF_0000_0000_0000; /// Used to isolate whether CC=12 is caused by the bandwidth dance or HID setup steps. const MINIMAL_INIT: bool = false; +/// Skip HCRST and use halt-resume instead. On Parallels, HCRST may destroy +/// the hypervisor's internal USB device model state. By just halting (RS=0), +/// swapping data structures, and resuming (RS=1), we preserve whatever +/// internal state the hypervisor built during UEFI firmware operation. +const SKIP_HCRST: bool = false; + /// Skip the bandwidth dance (StopEndpoint + re-ConfigureEndpoint per EP). /// Linux ftrace confirmed: Linux DOES perform the bandwidth dance for HID devices /// (3 ConfigureEndpoint commands total: 1 batch + Stop+re-ConfigEP per endpoint). @@ -58,23 +64,45 @@ const MINIMAL_INIT: bool = false; /// /// Set to true to test hypothesis that Parallels' virtual xHC internally rejects /// Perform Linux-style bandwidth dance (Stop EP + re-ConfigureEndpoint). -/// Parallels virtual xHC requires this sequence to actually enable interrupt -/// endpoints; skipping it leaves endpoints in a pseudo-running state that -/// returns CC=12 on the first interrupt TRB. -const SKIP_BW_DANCE: bool = false; +/// Tested both with and without — CC=12 persists regardless. +// EXPERIMENT: Skip BW dance. The Parallels hypervisor flags StopEndpoint +// commands as "not supported" and after 3 rapid StopEndpoints, triggers a +// cascading 2nd XHC controller reset that destroys endpoint state from 1st HCRST. +const SKIP_BW_DANCE: bool = true; + +/// UEFI-inherit mode: Instead of HCRST + full re-enumeration, inherit the +/// xHCI state left by UEFI firmware. We only swap out the command ring and +/// event ring, then use StopEndpoint + SetTRDequeuePointer to redirect +/// interrupt endpoints to our transfer rings. +/// +/// This tests whether UEFI-created endpoints are still functional after +/// ExitBootServices. If CC=12 disappears, the root cause is that the +/// Parallels hypervisor only creates internal endpoint state from UEFI's +/// ConfigureEndpoint commands, not from post-EBS re-enumeration. +const INHERIT_UEFI: bool = false; + +/// Experiment: after ConfigureEndpoint leaves EPs in Running state, explicitly +/// StopEndpoint + Set TR Dequeue Pointer before queueing the first Normal TRB. +/// This tests whether the xHC accepts the transfer ring address when set via +/// command (read from the command ring, which is proven to work) rather than +/// Deferred TRB queue: poll count at which to first queue interrupt TRBs. +/// 0 = queue immediately during init (current behavior). +/// 400 = defer to poll=400 (~2s after timer starts, matching Linux's msleep(2000)). +/// When > 0, start_hid_polling() is NOT called during init; instead, +/// poll_hid_events checks this value and queues TRBs on the first matching poll. +const DEFERRED_TRB_POLL: u64 = 0; + +/// Post-enumeration delay in milliseconds before first doorbell ring. +/// The Linux kernel module has msleep(2000) between ConfigureEndpoint and +/// the first interrupt TRB doorbell. This gives the Parallels virtual xHC +/// time to stabilize internal endpoint state. 0 = no delay. +const POST_ENUM_DELAY_MS: u32 = 0; /// Focus debug mode: only initialize the mouse device (slot=1), skip keyboard entirely. /// Reduces from 4 interrupt endpoints to 2, isolating whether CC=12 is caused by /// keyboard interference or is a fundamental per-endpoint issue. const MOUSE_ONLY: bool = false; -/// Send SET_PROTOCOL(Boot Protocol=0) to HID interfaces. -/// Linux's usbhid driver sends SET_PROTOCOL for boot keyboard (subclass=1, protocol=1) -/// during initial enumeration, but NOT during rebind (confirmed via usbmon on linux-probe VM). -/// Setting false matches the rebind sequence Linux uses. Testing whether SET_PROTOCOL -/// is causing Parallels to internally reset interrupt endpoints (producing CC=12). - - /// NEC XHCI vendor ID. pub const NEC_VENDOR_ID: u16 = 0x1033; /// NEC uPD720200 XHCI device ID. @@ -88,8 +116,8 @@ const MAX_SLOTS: usize = 32; /// Link TRB was using bit5 (IOC) instead of bit1 (TC), so the HC never /// toggled its cycle bit on wrap and stopped seeing post-wrap commands. const CMD_RING_SIZE: usize = 4096; -/// Event ring size in TRBs. -const EVENT_RING_SIZE: usize = 64; +/// Event ring size in TRBs (256 entries x 16 bytes = 4096 bytes, matching Linux). +const EVENT_RING_SIZE: usize = 256; /// Transfer ring size per endpoint in TRBs (last entry reserved for Link TRB). /// Larger transfer ring reduces the number of Stop EP + Set TR Dequeue resets. const TRANSFER_RING_SIZE: usize = 256; @@ -231,6 +259,9 @@ struct AlignedPage(T); /// /// Entry 0 is the scratchpad buffer array pointer (or 0 if not needed). /// Entries 1..MaxSlots are device context pointers. +/// +/// EXPERIMENT: Moved from .dma (NC at 0x50000000) to .bss (WB-cacheable, zeroed by boot.S). +/// Testing whether NC memory mapping causes CC=12 on Parallels. static mut DCBAA: AlignedPage<[u64; 256]> = AlignedPage([0u64; 256]); /// Command Ring: 64 TRBs x 16 bytes = 1KB. @@ -240,7 +271,7 @@ static mut CMD_RING_ENQUEUE: usize = 0; /// Command ring producer cycle state. static mut CMD_RING_CYCLE: bool = true; -/// Event Ring: 64 TRBs x 16 bytes = 1KB. +/// Event Ring: 256 TRBs x 16 bytes = 4KB (matches Linux xhci-ring.c). static mut EVENT_RING: Aligned64<[Trb; EVENT_RING_SIZE]> = Aligned64([Trb::zeroed(); EVENT_RING_SIZE]); /// Event ring dequeue pointer index. @@ -288,6 +319,10 @@ static mut RECONFIG_INPUT_CTX: [AlignedPage<[u8; 4096]>; MAX_SLOTS] = static mut DEVICE_CONTEXTS: [AlignedPage<[u8; 4096]>; MAX_SLOTS] = [const { AlignedPage([0u8; 4096]) }; MAX_SLOTS]; +/// Saved UEFI DCBAAP value, read before we overwrite it during init. +/// Used by INHERIT_UEFI to copy UEFI's device context data into our arrays. +static mut UEFI_DCBAAP_SAVED: u64 = 0; + /// HID report buffer for keyboard boot interface (8 bytes: modifier + reserved + 6 keycodes). static mut KBD_REPORT_BUF: Aligned64<[u8; 64]> = Aligned64([0u8; 64]); @@ -477,8 +512,27 @@ pub static DIAG_EP_STATE_AFTER_CC12: AtomicU32 = AtomicU32::new(0xFF); /// Diagnostic: endpoint output context state after NEC quirk + SetTRDeq reset (0xFF = not seen). /// Packed: slot<<16 | dci<<8 | state_bits[2:0]. Should be 1=Running if reset worked. pub static DIAG_EP_STATE_AFTER_RESET: AtomicU32 = AtomicU32::new(0xFF); +/// Diagnostic: raw EP context DWORDs at first CC=12 event (captured once). +/// DW0: EP State[2:0], Mult[9:8], MaxPStreams[14:10], Interval[23:16], MaxESITHi[31:24] +/// DW1: CErr[2:1], EPType[5:3], MaxBurst[15:8], MaxPacketSize[31:16] +/// DW2-3: TR Dequeue Pointer (64-bit) with DCS[0] +/// DW4: AvgTRBLen[15:0], MaxESITLo[31:16] +static DIAG_CC12_EP_DW0: AtomicU32 = AtomicU32::new(0xDEAD); +static DIAG_CC12_EP_DW1: AtomicU32 = AtomicU32::new(0xDEAD); +static DIAG_CC12_EP_DW2: AtomicU32 = AtomicU32::new(0xDEAD); +static DIAG_CC12_EP_DW3: AtomicU32 = AtomicU32::new(0xDEAD); +static DIAG_CC12_EP_DW4: AtomicU32 = AtomicU32::new(0xDEAD); +/// Diagnostic: Slot Context DW0 at first CC=12 (Context Entries in bits 31:27). +static DIAG_CC12_SLOT_DW0: AtomicU32 = AtomicU32::new(0xDEAD); +/// Diagnostic: Slot Context DW3 at first CC=12 (Slot State in bits 31:27, USB Addr in 7:0). +static DIAG_CC12_SLOT_DW3: AtomicU32 = AtomicU32::new(0xDEAD); +/// Diagnostic: DCBAA entry for the slot at first CC=12 (physical address of device context). +static DIAG_CC12_DCBAA: AtomicU64 = AtomicU64::new(0xDEAD); /// Diagnostic: MFINDEX register value (microframe index) for timing analysis. pub static DIAG_MFINDEX: AtomicU32 = AtomicU32::new(0); +/// Diagnostic: counts Transfer Events silently consumed during enumeration +/// by wait_for_event_inner (command_only mode) and control_transfer (non-EP0 skip). +pub static CONSUMED_XFER_DURING_ENUM: AtomicU64 = AtomicU64::new(0); /// Diagnostic: source of first queue_hid_transfer call. /// 0=unset, 1=inline init, 2=deferred poll=300, 3=reset_halted_endpoint, /// 4=MSI requeue, 5=CC=SUCCESS requeue, 6=poll CC=SUCCESS requeue @@ -514,7 +568,6 @@ static MOUSE_GET_REPORT_PENDING: AtomicBool = AtomicBool::new(false); /// to avoid CC=12 that occurs when TRBs are queued during initialization /// before the MSI pathway is active. Only set once; re-queue via MSI/error handlers. static KBD_TRB_FIRST_QUEUED: AtomicBool = AtomicBool::new(false); -static DEFERRED_RECFG_DONE: AtomicBool = AtomicBool::new(false); /// Whether initial HID interrupt TRBs have been queued post-init. /// TRBs are deferred until after XHCI_INITIALIZED and SPI enable so the full @@ -547,6 +600,10 @@ fn virt_to_phys(virt: u64) -> u64 { /// /// Must be called after writing DMA descriptors/data and before issuing /// DMA commands, so the device sees the updated data in physical memory. +/// +/// RE-ENABLED: DMA structures are now in .bss (WB-cacheable memory). +/// CPU stores go to cache and must be flushed to memory before the +/// xHCI controller (via hypervisor DMA emulation) can see them. #[inline] fn dma_cache_clean(ptr: *const u8, len: usize) { const CACHE_LINE: usize = 64; @@ -558,14 +615,18 @@ fn dma_cache_clean(ptr: *const u8, len: usize) { } } unsafe { - core::arch::asm!("dsb sy", options(nostack, preserves_flags)); + core::arch::asm!("dsb st", options(nostack, preserves_flags)); } } /// Invalidate a range of memory in CPU caches after a device DMA write. /// -/// Must be called after a DMA read completes and before the CPU reads -/// the DMA buffer, to ensure the CPU sees the device-written data. +/// Must be called after the xHCI controller writes to DMA memory (e.g., +/// output device contexts, event ring), before the CPU reads the data. +/// +/// RE-ENABLED: DMA structures are now in .bss (WB-cacheable memory). +/// Uses dc civac (clean+invalidate) which first writes back any dirty +/// data then invalidates, ensuring the CPU reads fresh data from memory. #[inline] fn dma_cache_invalidate(ptr: *const u8, len: usize) { const CACHE_LINE: usize = 64; @@ -577,7 +638,7 @@ fn dma_cache_invalidate(ptr: *const u8, len: usize) { } } unsafe { - core::arch::asm!("dsb sy", options(nostack, preserves_flags)); + core::arch::asm!("dsb ld", options(nostack, preserves_flags)); } } @@ -626,6 +687,26 @@ struct XhciTraceRecord { data_len: u32, } +// ============================================================================= +// MMIO Write Trace — captures every write32() from HCRST through first doorbell +// ============================================================================= + +/// Maximum entries in the MMIO write trace buffer. +const MMIO_TRACE_MAX: usize = 4096; + +/// Whether MMIO write tracing is active. +static MMIO_TRACE_ACTIVE: AtomicBool = AtomicBool::new(false); + +/// Current write index into MMIO_TRACE_BUF. +static MMIO_TRACE_IDX: AtomicU32 = AtomicU32::new(0); + +/// BAR base virtual address, set once during init so write32 can compute offsets. +static MMIO_TRACE_BAR_BASE: AtomicU64 = AtomicU64::new(0); + +/// Trace buffer: (offset_from_BAR, value) pairs. +/// offset and value are each u32, packed into a u64 for atomic-free static init. +static mut MMIO_TRACE_BUF: [(u32, u32); MMIO_TRACE_MAX] = [(0u32, 0u32); MMIO_TRACE_MAX]; + /// Whether tracing is active. static XHCI_TRACE_ACTIVE: AtomicBool = AtomicBool::new(false); /// Monotonic sequence number for trace records. @@ -817,6 +898,148 @@ fn xhci_trace_cache_op(addr: u64, len: u32) { xhci_trace(XhciTraceOp::CacheOp, 0, 0, &buf); } +/// Format the xHCI trace buffer as a String for procfs consumption. +/// Same data as xhci_trace_dump() but writes to a String instead of serial. +pub fn format_trace_buffer() -> alloc::string::String { + use alloc::string::String; + use core::fmt::Write; + + let mut out = String::new(); + let total = XHCI_TRACE_SEQ.load(Ordering::Relaxed); + if total == 0 { + let _ = writeln!(out, "=== XHCI_TRACE_START ==="); + let _ = writeln!(out, "(no records)"); + let _ = writeln!(out, "=== XHCI_TRACE_END ==="); + return out; + } + + let start = if total as usize <= XHCI_TRACE_MAX_RECORDS { + 0u32 + } else { + total - XHCI_TRACE_MAX_RECORDS as u32 + }; + + let _ = writeln!(out, "=== XHCI_TRACE_START total={} ===", total); + + for seq in start..total { + let idx = seq as usize % XHCI_TRACE_MAX_RECORDS; + let rec = unsafe { + &*core::ptr::addr_of!(XHCI_TRACE_RECORDS) + .cast::() + .add(idx) + }; + + let op_name = op_name_str(rec.op); + + let _ = writeln!( + out, + "T {:04} {:12} S={:02} E={:02} TS={:016X} LEN={:04X}", + rec.seq, op_name, rec.slot, rec.dci, rec.timestamp, rec.data_len, + ); + + if rec.data_len > 0 && rec.data_offset != 0xFFFF_FFFF { + let off = rec.data_offset as usize; + let len = rec.data_len as usize; + if off + len <= XHCI_TRACE_DATA_SIZE { + let data = unsafe { + core::slice::from_raw_parts( + core::ptr::addr_of!(XHCI_TRACE_DATA) + .cast::() + .add(off), + len, + ) + }; + + if rec.op == 50 { + if let Ok(s) = core::str::from_utf8(data) { + let _ = writeln!(out, " \"{}\"", s); + } + continue; + } + + let mut i = 0; + while i < len { + let row_end = (i + 16).min(len); + let _ = write!(out, " "); + let mut j = i; + while j < row_end { + let dw_end = (j + 4).min(row_end); + let mut k = j; + while k < dw_end { + let byte = data[k]; + let _ = write!(out, "{:02X}", byte); + k += 1; + } + if dw_end < row_end { + let _ = write!(out, " "); + } + j = dw_end; + } + let _ = writeln!(out); + i += 16; + } + } + } + } + + let _ = writeln!(out, "=== XHCI_TRACE_END ==="); + + // Append diagnostic counters for btrace to parse + let _ = writeln!(out, "=== XHCI_DIAG ==="); + let _ = writeln!(out, "poll_count {}", POLL_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "event_count {}", EVENT_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "consumed_xfer_enum {}", CONSUMED_XFER_DURING_ENUM.load(Ordering::Relaxed)); + let _ = writeln!(out, "first_xfer_cc {}", DIAG_FIRST_XFER_CC.load(Ordering::Relaxed)); + let _ = writeln!(out, "first_queue_src {}", DIAG_FIRST_QUEUE_SOURCE.load(Ordering::Relaxed)); + let _ = writeln!(out, "ep_state_cc12 {}", DIAG_EP_STATE_AFTER_CC12.load(Ordering::Relaxed)); + let _ = writeln!(out, "endpoint_resets {}", ENDPOINT_RESET_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "xfer_other {}", XFER_OTHER_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "ep_before_db 0x{:08x}", DIAG_EP_STATE_BEFORE_DB.load(Ordering::Relaxed)); + let _ = writeln!(out, "slot_before_db 0x{:08x}", DIAG_SLOT_STATE_BEFORE_DB.load(Ordering::Relaxed)); + let _ = writeln!(out, "trdp_output 0x{:016x}", DIAG_TRDP_FROM_OUTPUT.load(Ordering::Relaxed)); + let _ = writeln!(out, "portsc_before_db 0x{:08x}", DIAG_PORTSC_BEFORE_DB.load(Ordering::Relaxed)); + let _ = writeln!(out, "first_queued_phys 0x{:016x}", DIAG_FIRST_QUEUED_PHYS.load(Ordering::Relaxed)); + let _ = writeln!(out, "first_xfer_ptr 0x{:016x}", DIAG_FIRST_XFER_PTR.load(Ordering::Relaxed)); + let slep = DIAG_FIRST_XFER_SLEP.load(Ordering::Relaxed); + let _ = writeln!(out, "first_xfer_slep slot={} ep={}", (slep >> 8) & 0xFF, slep & 0xFF); + let _ = writeln!(out, "first_xfer_status 0x{:08x}", DIAG_FIRST_XFER_STATUS.load(Ordering::Relaxed)); + let _ = writeln!(out, "first_xfer_control 0x{:08x}", DIAG_FIRST_XFER_CONTROL.load(Ordering::Relaxed)); + let _ = writeln!(out, "ep_state_after_reset 0x{:08x}", DIAG_EP_STATE_AFTER_RESET.load(Ordering::Relaxed)); + let _ = writeln!(out, "reset_fail_count {}", ENDPOINT_RESET_FAIL_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "xo_err_count {}", XO_ERR_COUNT.load(Ordering::Relaxed)); + let _ = writeln!(out, "xo_last_info slot={} ep={} cc={}", + (XO_LAST_INFO.load(Ordering::Relaxed) >> 16) & 0xFF, + (XO_LAST_INFO.load(Ordering::Relaxed) >> 8) & 0xFF, + XO_LAST_INFO.load(Ordering::Relaxed) & 0xFF); + let _ = writeln!(out, "skip_bw_dance {}", SKIP_BW_DANCE); + let _ = writeln!(out, "=== XHCI_DIAG_END ==="); + + out +} + +/// Map an operation byte to its human-readable name. +fn op_name_str(op: u8) -> &'static str { + match op { + 1 => "MMIO_W32", + 2 => "MMIO_W64", + 3 => "MMIO_R32", + 10 => "CMD_SUBMIT", + 11 => "CMD_COMPLETE", + 12 => "XFER_SUBMIT", + 13 => "XFER_EVENT", + 14 => "DOORBELL", + 20 => "INPUT_CTX", + 21 => "OUTPUT_CTX", + 22 => "XFER_RING_SETUP", + 30 => "CACHE_OP", + 31 => "SET_TR_DEQ", + 40 => "EP_STATE", + 41 => "PORT_SC", + 50 => "NOTE", + _ => "UNKNOWN", + } +} + /// Dump the xHCI trace buffer to serial in a parseable hex format. /// Called once after init completes. Uses serial_println which is fine post-init. #[allow(dead_code)] @@ -846,26 +1069,7 @@ fn xhci_trace_dump() { .add(idx) }; - // Op name for readability - let op_name = match rec.op { - 1 => "MMIO_W32", - 2 => "MMIO_W64", - 3 => "MMIO_R32", - 10 => "CMD_SUBMIT", - 11 => "CMD_COMPLETE", - 12 => "XFER_SUBMIT", - 13 => "XFER_EVENT", - 14 => "DOORBELL", - 20 => "INPUT_CTX", - 21 => "OUTPUT_CTX", - 22 => "XFER_RING_SETUP", - 30 => "CACHE_OP", - 31 => "SET_TR_DEQ", - 40 => "EP_STATE", - 41 => "PORT_SC", - 50 => "NOTE", - _ => "UNKNOWN", - }; + let op_name = op_name_str(rec.op); crate::serial_println!( "T {:04} {:12} S={:02} E={:02} TS={:016X} LEN={:04X}", @@ -939,29 +1143,126 @@ fn xhci_trace_dump() { crate::serial_println!("=== XHCI_TRACE_END ==="); } +// ============================================================================= +// Milestone-based initialization instrumentation (matches Linux module) +// +// M1: CONTROLLER_DISCOVERY — BAR mapped, capabilities read +// M2: CONTROLLER_RESET — HCRST done, CNR clear, HCH=1 +// M3: DATA_STRUCTURES — DCBAA, CMD ring, EVT ring, ERST programmed +// M4: CONTROLLER_RUNNING — RS=1, INTE=1, IMAN.IE=1 +// M5: PORT_DETECTION — Connected ports identified, speed known +// M6: SLOT_ENABLE — EnableSlot CC=1, slot ID allocated +// M7: DEVICE_ADDRESS — Input ctx built, AddressDevice CC=1 +// M8: ENDPOINT_CONFIG — ConfigureEndpoint CC=1, BW dance done +// M9: HID_CLASS_SETUP — SET_CONFIGURATION, SET_IDLE, descriptors +// M10: INTERRUPT_TRANSFER — Normal TRBs queued, doorbells rung +// M11: EVENT_DELIVERY — First transfer event (HID data received) +// ============================================================================= + +// Milestone constants — intentionally kept for re-enablement of tracing macros. +#[allow(dead_code)] const M_DISCOVERY: u8 = 1; +#[allow(dead_code)] const M_RESET: u8 = 2; +#[allow(dead_code)] const M_DATA_STRUC: u8 = 3; +#[allow(dead_code)] const M_RUNNING: u8 = 4; +#[allow(dead_code)] const M_PORT_DET: u8 = 5; +#[allow(dead_code)] const M_SLOT_EN: u8 = 6; +#[allow(dead_code)] const M_ADDR_DEV: u8 = 7; +#[allow(dead_code)] const M_EP_CONFIG: u8 = 8; +#[allow(dead_code)] const M_HID_SETUP: u8 = 9; +#[allow(dead_code)] const M_INTR_XFER: u8 = 10; +#[allow(dead_code)] const M_EVT_DELIV: u8 = 11; +#[allow(dead_code)] const M_TOTAL: u8 = 11; + +#[allow(dead_code)] +const MILESTONE_NAMES: [&str; 12] = [ + "UNUSED", + "CONTROLLER_DISCOVERY", + "CONTROLLER_RESET", + "DATA_STRUCTURES", + "CONTROLLER_RUNNING", + "PORT_DETECTION", + "SLOT_ENABLE", + "DEVICE_ADDRESS", + "ENDPOINT_CONFIG", + "HID_CLASS_SETUP", + "INTERRUPT_TRANSFER", + "EVENT_DELIVERY", +]; + +macro_rules! ms_begin { ($m:expr) => {}; } +macro_rules! ms_pass { ($m:expr) => {}; } +macro_rules! ms_fail { ($m:expr, $reason:expr) => {}; } +macro_rules! ms_kv { ($m:expr, $fmt:literal $(, $arg:expr)*) => {}; } + +/// Hex-dump a DMA buffer under a milestone (silenced). +#[allow(dead_code)] +fn ms_dump(_m: u8, _label: &str, _phys: u64, _buf: *const u8, _len: usize) {} + +/// Dump a TRB under a milestone (silenced). +fn ms_trb(_m: u8, _label: &str, _trb: &Trb) {} + +/// Dump all key xHCI registers under a milestone (silenced). +fn ms_regs(_m: u8, _op_base: u64, _ir0_base: u64) {} + // ============================================================================= // MMIO Register Access // ============================================================================= #[inline] fn read32(addr: u64) -> u32 { - unsafe { core::ptr::read_volatile(addr as *const u32) } + unsafe { + let val = core::ptr::read_volatile(addr as *const u32); + // DSB to ensure the read completes before subsequent operations, + // matching Linux's readl() which includes a post-read barrier. + core::arch::asm!("dsb ld", options(nostack, preserves_flags)); + val + } } #[inline] fn write32(addr: u64, val: u32) { - unsafe { core::ptr::write_volatile(addr as *mut u32, val) } + // Record to MMIO trace buffer if tracing is active + if MMIO_TRACE_ACTIVE.load(Ordering::Relaxed) { + let bar_base = MMIO_TRACE_BAR_BASE.load(Ordering::Relaxed); + if bar_base != 0 && addr >= bar_base { + let offset = (addr - bar_base) as u32; + let idx = MMIO_TRACE_IDX.fetch_add(1, Ordering::Relaxed) as usize; + if idx < MMIO_TRACE_MAX { + unsafe { + MMIO_TRACE_BUF[idx] = (offset, val); + } + } + } + } + unsafe { + // DSB before write ensures all prior Normal memory writes (e.g. TRB data) + // are globally visible before this Device-nGnRnE MMIO write. + // DSB after write ensures this MMIO write reaches the device before + // subsequent operations. Matches Linux's writel() barrier semantics. + core::arch::asm!("dsb st", options(nostack, preserves_flags)); + core::ptr::write_volatile(addr as *mut u32, val); + core::arch::asm!("dsb st", options(nostack, preserves_flags)); + } } #[inline] #[allow(dead_code)] // Part of MMIO register access API fn read64(addr: u64) -> u64 { - unsafe { core::ptr::read_volatile(addr as *const u64) } + // xHCI spec: 64-bit registers must be accessed as two 32-bit reads (lo then hi). + // Parallels' MMIO trap handler may not correctly handle a single 64-bit load. + // This matches Linux's xhci_read64() which uses two readl() calls. + let lo = read32(addr) as u64; + let hi = read32(addr + 4) as u64; + (hi << 32) | lo } #[inline] fn write64(addr: u64, val: u64) { - unsafe { core::ptr::write_volatile(addr as *mut u64, val) } + // xHCI spec: 64-bit registers must be accessed as two 32-bit writes (lo then hi). + // Parallels' MMIO trap handler may not correctly handle a single 64-bit store. + // This matches Linux's xhci_write64() which uses two writel() calls. + write32(addr, val as u32); + write32(addr + 4, (val >> 32) as u32); } // ============================================================================= @@ -979,6 +1280,41 @@ fn wait_for bool>(f: F, max_iters: u32) -> Result<(), &'static str> { Err("XHCI timeout") } +/// Busy-wait for `ms` milliseconds using the ARM64 generic timer. +/// Falls back to a spin loop if the timer frequency is unavailable. +fn delay_ms(ms: u32) { + let freq: u64; + unsafe { core::arch::asm!("mrs {}, cntfrq_el0", out(reg) freq) }; + if freq == 0 { + // Fallback: ~1ms per iteration at ~1GHz + for _ in 0..ms { + for _ in 0..200_000u32 { + core::hint::spin_loop(); + } + } + return; + } + let start: u64; + unsafe { core::arch::asm!("mrs {}, cntpct_el0", out(reg) start) }; + let ticks = (freq / 1000) * ms as u64; + loop { + let now: u64; + unsafe { core::arch::asm!("mrs {}, cntpct_el0", out(reg) now) }; + if now.wrapping_sub(start) >= ticks { + break; + } + core::hint::spin_loop(); + } +} + +// ============================================================================= +// EHCI Controller Reset (Parallels workaround) +// ============================================================================= + +/// Reset the EHCI controller at PCI 00:02.0 [8086:265c]. +/// +/// The Parallels hypervisor's virtual USB subsystem requires the EHCI controller +/// to be reset in close proximity to the xHCI reset for the virtual xHC to // ============================================================================= // Command Ring Operations // ============================================================================= @@ -1046,11 +1382,10 @@ fn enqueue_command(trb: Trb) { /// Slot 0, target 0 = host controller command ring. /// Slot N, target DCI = endpoint for that device slot. fn ring_doorbell(state: &XhciState, slot: u8, target: u8) { + // write32() now includes pre- and post-DSB barriers, + // ensuring TRB data is globally visible before the doorbell + // and the doorbell reaches the device before continuing. write32(state.db_base + (slot as u64) * 4, target as u32); - // DSB to ensure the doorbell write reaches the device - unsafe { - core::arch::asm!("dsb sy", options(nostack, preserves_flags)); - } } /// Wait for an event on the event ring, with timeout. @@ -1093,17 +1428,11 @@ fn wait_for_event_inner(state: &XhciState, command_only: bool) -> Result Result Result { // Slot and Device Commands // ============================================================================= +/// Issue a No-Op command and wait for completion. Used to warm up the command +/// ring after RS=1, matching Linux xhci_hcd which sends a NEC vendor NOOP as +/// its first command before Enable Slot. +fn send_noop(state: &XhciState) -> Result<(), &'static str> { + let trb = Trb { + param: 0, + status: 0, + control: trb_type::NOOP << 10, + }; + enqueue_command(trb); + ring_doorbell(state, 0, 0); + + let event = wait_for_command(state)?; + let cc = event.completion_code(); + if cc != completion_code::SUCCESS { + xhci_trace_note(0, "err:noop"); + return Err("XHCI NOOP failed"); + } + xhci_trace_note(0, "noop_ok"); + Ok(()) +} + /// Issue an Enable Slot command and return the assigned slot ID. fn enable_slot(state: &XhciState) -> Result { let trb = Trb { @@ -1166,12 +1518,12 @@ fn enable_slot(state: &XhciState) -> Result { let event = wait_for_command(state)?; let cc = event.completion_code(); + let slot_id = event.slot_id(); if cc != completion_code::SUCCESS { xhci_trace_note(0, "err:enable_slot"); return Err("XHCI EnableSlot failed"); } - let slot_id = event.slot_id(); xhci_trace_note(slot_id, "enable_slot"); Ok(slot_id) } @@ -1293,21 +1645,31 @@ fn address_device(state: &XhciState, slot_id: u8, port_id: u8) -> Result<(), &'s // Note: the ftrace agent misidentified the cycle bit (b:C) as the BSR bit — // BSR=1 causes CC=19 (Context State Error) on ConfigureEndpoint. let input_ctx_phys = virt_to_phys(&raw const INPUT_CONTEXTS[slot_idx] as u64); + + ms_kv!(M_ADDR_DEV, "slot={} port={} speed={}", slot_id, port_id, port_speed); + ms_dump(M_ADDR_DEV, "input_ctx", input_ctx_phys, input_base, ctx_size * 3); + let trb = Trb { param: input_ctx_phys, status: 0, // AddressDevice type, Slot ID in bits 31:24 control: (trb_type::ADDRESS_DEVICE << 10) | ((slot_id as u32) << 24), }; + ms_trb(M_ADDR_DEV, "AddressDevice_TRB", &trb); enqueue_command(trb); ring_doorbell(state, 0, 0); let event = wait_for_command(state)?; let cc = event.completion_code(); + ms_kv!(M_ADDR_DEV, "CC={} slot={}", cc, slot_id); if cc != completion_code::SUCCESS { return Err("XHCI AddressDevice failed"); } + // Dump output context after successful AddressDevice + dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); + ms_dump(M_ADDR_DEV, "output_ctx", dev_ctx_phys, (*dev_ctx).0.as_ptr(), ctx_size * 2); + xhci_trace_note(slot_id, "address_device"); Ok(()) } @@ -1475,9 +1837,16 @@ fn control_transfer( enqueue_transfer(slot_idx, data_trb); } - // Status Stage TRB - // Direction is opposite of data stage (or IN if no data stage) - let status_dir: u32 = if data_len == 0 || direction_in { 0 } else { 1 << 16 }; + // Status Stage TRB — xHCI spec Section 4.11.2.2: + // No Data Stage (TRT=0): Status direction = IN (bit 16 = 1) + // IN Data Stage (TRT=3): Status direction = OUT (bit 16 = 0) + // OUT Data Stage (TRT=2): Status direction = IN (bit 16 = 1) + // i.e. Status direction is always opposite of data direction, defaulting to IN. + let status_dir: u32 = if data_len > 0 && direction_in { + 0 // IN data stage → Status direction OUT + } else { + 1 << 16 // No data stage or OUT data stage → Status direction IN + }; let status_trb = Trb { param: 0, status: 0, @@ -1513,6 +1882,7 @@ fn control_transfer( } // Not our EP0 event — stale interrupt endpoint event, skip it + CONSUMED_XFER_DURING_ENUM.fetch_add(1, Ordering::Relaxed); continue; } @@ -1582,28 +1952,6 @@ fn get_device_descriptor( Ok(()) } -/// Send USB 3.0 SET_ISOCH_DELAY request (bRequest=0x31). -/// -/// Linux sends this immediately after the first 8-byte device descriptor read. -/// wValue = isochronous delay in nanoseconds (Linux uses 40ns = 0x0028). -/// This is a USB 3.0 standard request that may be required by the Parallels -/// virtual xHC for proper endpoint activation. -fn set_isoch_delay( - state: &XhciState, - slot_id: u8, -) -> Result<(), &'static str> { - let setup = SetupPacket { - bm_request_type: 0x00, // Host-to-device, Standard, Device - b_request: request::SET_ISOCH_DELAY, - w_value: 0x0028, // 40 nanoseconds (matches Linux) - w_index: 0, - w_length: 0, - }; - - control_transfer(state, slot_id, &setup, 0, 0, false)?; - Ok(()) -} - /// Read string descriptors from the device, matching Linux's enumeration. /// /// Linux reads string descriptors #0 (language IDs), #2 (Product), #1 (Manufacturer), @@ -1651,6 +1999,28 @@ fn read_string_descriptors( } } +/// Send SET_ISOCH_DELAY to a USB 3.0 device. +/// +/// This is a standard USB 3.0 request (bRequest=0x1F) that sets the isochronous +/// delay to 40us (wValue=0x0028). Linux xhci_hcd sends this after the short +/// (8-byte) device descriptor read, before reading the full descriptor. +#[allow(dead_code)] +fn set_isoch_delay( + state: &XhciState, + slot_id: u8, +) -> Result<(), &'static str> { + let setup = SetupPacket { + bm_request_type: 0x00, // Host-to-device, Standard, Device + b_request: 0x1F, // SET_ISOCH_DELAY + w_value: 0x0028, // 40us delay + w_index: 0, + w_length: 0, + }; + // No data stage — just Setup + Status + control_transfer(state, slot_id, &setup, 0, 0, false)?; + Ok(()) +} + /// Read the BOS (Binary Object Store) descriptor from a USB 3.0 device. /// /// Linux reads this after the full device descriptor and before the config descriptor. @@ -1782,8 +2152,6 @@ fn set_configuration( /// Linux's USB core sends SET_INTERFACE(alt=0) for each interface during driver /// probe. Parallels' virtual USB device model may require this to activate the /// interface's interrupt endpoints. -/// NOTE: Tested and causes system hang on Parallels vxHC for HID devices. -#[allow(dead_code)] fn set_interface( state: &XhciState, slot_id: u8, @@ -1867,6 +2235,29 @@ fn set_idle( Ok(()) } +/// Send SET_PROTOCOL to a HID boot-class interface. +/// +/// Boot-class HID devices (subclass=1) require SET_PROTOCOL to activate +/// their interrupt endpoints for the specified protocol mode. The Linux kernel +/// module sends this for all boot devices; omitting it may cause the Parallels +/// vxHC to not deliver interrupt transfers. +fn set_protocol( + state: &XhciState, + slot_id: u8, + interface: u8, + protocol: u8, // 0 = Boot, 1 = Report +) -> Result<(), &'static str> { + let setup = SetupPacket { + bm_request_type: 0x21, // Host-to-device, class, interface + b_request: hid_request::SET_PROTOCOL, + w_value: protocol as u16, + w_index: interface as u16, + w_length: 0, + }; + control_transfer(state, slot_id, &setup, 0, 0, false)?; + Ok(()) +} + /// Send SET_REPORT(Output) to clear LED indicators on a HID keyboard interface. /// /// Linux sends this during keyboard init: SET_REPORT(Output, Report ID=0, data=0x00). @@ -1996,7 +2387,7 @@ struct PendingEp { b_interval: u8, ss_max_burst: u8, ss_bytes_per_interval: u16, - ss_mult: u8, + _ss_mult: u8, } /// Configure all HID interrupt endpoints in a single ConfigureEndpoint command. @@ -2023,6 +2414,10 @@ fn configure_endpoints_batch( return Ok(()); } + // --- M8: ENDPOINT_CONFIG --- + ms_begin!(M_EP_CONFIG); + ms_kv!(M_EP_CONFIG, "slot={} ep_count={}", slot_id, ep_count); + let slot_idx = (slot_id - 1) as usize; let ctx_size = state.context_size; @@ -2197,6 +2592,9 @@ fn configure_endpoints_batch( // Issue batch ConfigureEndpoint using INPUT_CONTEXTS directly. let input_ctx_phys = virt_to_phys(&raw const INPUT_CONTEXTS[slot_idx] as u64); + ms_kv!(M_EP_CONFIG, "slot={} ep_count={} add_flags=0x{:x} max_dci={}", + slot_id, ep_count, add_flags, max_dci); + ms_dump(M_EP_CONFIG, "input_ctx", input_ctx_phys, input_base, (1 + max_dci as usize + 1) * ctx_size); { xhci_trace_input_ctx(slot_id, input_base, ctx_size, max_dci as u8); let trb = Trb { @@ -2204,6 +2602,7 @@ fn configure_endpoints_batch( status: 0, control: (trb_type::CONFIGURE_ENDPOINT << 10) | ((slot_id as u32) << 24), }; + ms_trb(M_EP_CONFIG, "ConfigureEndpoint_TRB", &trb); xhci_trace_trb(XhciTraceOp::CommandSubmit, slot_id, 0, &trb); enqueue_command(trb); ring_doorbell(state, 0, 0); @@ -2211,9 +2610,19 @@ fn configure_endpoints_batch( let event = wait_for_command(state)?; xhci_trace_trb(XhciTraceOp::CommandComplete, slot_id, 0, &event); let cc = event.completion_code(); + ms_kv!(M_EP_CONFIG, "ConfigureEndpoint slot={} CC={} add_flags=0x{:x}", + slot_id, cc, add_flags); if cc != completion_code::SUCCESS { + ms_fail!(M_EP_CONFIG, "ConfigureEndpoint command failed"); return Err("XHCI ConfigureEndpoint failed"); } + + // Dump output device context after successful ConfigureEndpoint + dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); + let out_ctx_len = (1 + max_dci as usize) * ctx_size; + let dev_ctx_phys = virt_to_phys(&raw const DEVICE_CONTEXTS[slot_idx] as u64); + ms_dump(M_EP_CONFIG, "output_ctx_post_cfgep", dev_ctx_phys, + (*dev_ctx).0.as_ptr(), out_ctx_len); } // Bandwidth dance: StopEndpoint + re-ConfigureEndpoint per endpoint. @@ -2235,6 +2644,7 @@ fn configure_endpoints_batch( // the same endpoint contexts. Only the Slot Context is refreshed from // the Output Context. if !SKIP_BW_DANCE { + ms_kv!(M_EP_CONFIG, "BW_dance: slot={} ep_count={}", slot_id, ep_count); // BW dance: StopEndpoint + re-ConfigureEndpoint per endpoint. // // Linux reuses ONE re-ConfigEP Input Context (distinct from the initial @@ -2249,6 +2659,7 @@ fn configure_endpoints_batch( if let Some(ref ep) = endpoints[i] { let dci = ep.dci; + ms_kv!(M_EP_CONFIG, "BW_stop: slot={} dci={}", slot_id, dci); // Step 1: Stop Endpoint let stop_trb = Trb { param: 0, @@ -2263,6 +2674,7 @@ fn configure_endpoints_batch( let stop_event = wait_for_command(state)?; xhci_trace_trb(XhciTraceOp::CommandComplete, slot_id, dci, &stop_event); let _stop_cc = stop_event.completion_code(); + ms_kv!(M_EP_CONFIG, "StopEndpoint slot={} dci={} CC={}", slot_id, dci, _stop_cc); // Read output context EP state after StopEP (should be 3=Stopped) dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); let _stop_ep_state = core::ptr::read_volatile( @@ -2276,7 +2688,10 @@ fn configure_endpoints_batch( // then Re-ConfigureEndpoint using that buffer. core::ptr::write_bytes(reconfig_base as *mut u8, 0, 4096); - // ICC: Drop=0, Add=A0 + all endpoints (same as initial). + // ICC: Drop=0, Add=ALL endpoints (same as initial ConfigEP). + // Linux's BW dance uses the FULL add_flags (SLOT + all DCIs) + // for every re-ConfigureEndpoint, not just the stopped one. + // The Linux module code: rctrl->add_flags = add_flags; core::ptr::write_volatile(reconfig_base as *mut u32, 0u32); core::ptr::write_volatile(reconfig_base.add(4) as *mut u32, add_flags); @@ -2298,13 +2713,15 @@ fn configure_endpoints_batch( (rc_slot_dw0 & !(0x1F << 27)) | (max_dci << 27), ); - // Endpoint contexts: copy all endpoints from Output Context. + // Endpoint context: copy ALL endpoint contexts from Output Context. + // Linux's BW dance copies every endpoint (not just the stopped one), + // clearing EP state bits (DW0 bits 2:0) for each. for j in 0..ep_count { - if let Some(ref epj) = endpoints[j] { - let ep_dci = epj.dci as usize; + if let Some(ref ep_j) = endpoints[j] { + let ep_dci = ep_j.dci as usize; let rc_ep = reconfig_base.add((1 + ep_dci) * ctx_size); let src_ep = (*dev_ctx).0.as_ptr().add(ep_dci * ctx_size); - for dw_offset in (0..20usize).step_by(4) { + for dw_offset in (0..32usize).step_by(4) { let val = core::ptr::read_volatile( src_ep.add(dw_offset) as *const u32, ); @@ -2333,6 +2750,7 @@ fn configure_endpoints_batch( let reconfig_event = wait_for_command(state)?; xhci_trace_trb(XhciTraceOp::CommandComplete, slot_id, 0, &reconfig_event); let _reconfig_cc = reconfig_event.completion_code(); + ms_kv!(M_EP_CONFIG, "BW_dance slot={} dci={} CC={}", slot_id, dci, _reconfig_cc); // Diagnostic: verify TR Dequeue pointer in output context after re-ConfigEP. dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); @@ -2344,6 +2762,12 @@ fn configure_endpoints_batch( &raw const TRANSFER_RINGS[HID_RING_BASE + ep.hid_idx] as u64 ); xhci_trace_output_ctx(slot_id, (*dev_ctx).0.as_ptr(), ctx_size, max_dci as u8); + + // Dump output device context after BW dance StopEndpoint+Reconfigure for this DCI + let dev_ctx_phys_bw = virt_to_phys(&raw const DEVICE_CONTEXTS[slot_idx] as u64); + let out_ctx_len_bw = (1 + max_dci as usize) * ctx_size; + ms_dump(M_EP_CONFIG, "output_ctx_post_bw_dance", dev_ctx_phys_bw, + (*dev_ctx).0.as_ptr(), out_ctx_len_bw); } } } @@ -2371,6 +2795,7 @@ fn configure_endpoints_batch( } } + ms_pass!(M_EP_CONFIG); Ok(()) } @@ -2409,6 +2834,7 @@ fn configure_hid( interface_number: u8, is_keyboard: bool, is_nkro: bool, // Non-boot HID interface (subclass=0, Report ID protocol) + is_boot: bool, // Boot-class interface (subclass=1) hid_idx: usize, // Transfer ring index (0=kbd boot, 1=mouse, 2=kbd NKRO, 3=mouse2) dci: u8, hid_report_len: u16, // wDescriptorLength from HID descriptor (0 = unknown) @@ -2499,7 +2925,7 @@ fn configure_hid( // immediately following this endpoint descriptor let mut ss_max_burst: u8 = 0; let mut ss_bytes_per_interval: u16 = 0; - let mut ss_mult: u8 = 0; + let mut _ss_mult: u8 = 0; let ss_offset = ep_offset + ep_len; if ss_offset + 2 <= config_len { let ss_len = config_buf[ss_offset] as usize; @@ -2507,7 +2933,7 @@ fn configure_hid( if ss_type == 0x30 && ss_len >= 6 && ss_offset + ss_len <= config_len { ss_max_burst = config_buf[ss_offset + 2]; // bmAttributes bits[1:0] = Mult (max burst multiplier) - ss_mult = config_buf[ss_offset + 3] & 0x3; + _ss_mult = config_buf[ss_offset + 3] & 0x3; ss_bytes_per_interval = u16::from_le_bytes([ config_buf[ss_offset + 4], config_buf[ss_offset + 5], @@ -2565,7 +2991,7 @@ fn configure_hid( b_interval: ep_desc.b_interval, ss_max_burst, ss_bytes_per_interval, - ss_mult, + _ss_mult, }); if dci > max_dci { max_dci = dci; @@ -2579,6 +3005,7 @@ fn configure_hid( interface_number: iface.b_interface_number, is_keyboard, is_nkro, + is_boot: iface.b_interface_sub_class == 1, hid_idx, dci, hid_report_len, @@ -2604,17 +3031,19 @@ fn configure_hid( } // ========================================================================= - // Phase 1b: ConfigureEndpoint BEFORE SET_CONFIGURATION (matching Linux) + // Phase 1b: ConfigureEndpoint BEFORE SET_CONFIGURATION (matching Linux module) // - // Linux's xhci_check_bandwidth() issues ConfigureEndpoint (+ BW dance) - // BEFORE usb_set_configuration() sends the USB SET_CONFIGURATION request. - // This ensures the xHC has transfer rings ready before the device - // activates its endpoints. The C harness uses this order and avoids CC=12. + // Both spec-correct (SET_CONFIG first) and module order (ConfigureEndpoint first) + // produce CC=12 on Parallels. Keeping module order for consistency with Linux module. // ========================================================================= if ep_count > 0 { configure_endpoints_batch(state, slot_id, &pending_eps, ep_count)?; } + // --- M9: HID_CLASS_SETUP --- + ms_begin!(M_HID_SETUP); + ms_kv!(M_HID_SETUP, "SET_CONFIGURATION slot={} config={}", slot_id, config_value); + set_configuration(state, slot_id, config_value)?; // Diagnostic: dump endpoint states AFTER ConfigureEndpoint @@ -2628,28 +3057,57 @@ fn configure_hid( if max_dci != 0 { xhci_trace_output_ctx(slot_id, (*dev_ctx).0.as_ptr(), ctx_size, max_dci); } + + // Dump output device context after SET_CONFIGURATION + let dev_ctx_phys = virt_to_phys(&raw const DEVICE_CONTEXTS[slot_idx] as u64); + let out_ctx_len = (1 + max_dci as usize) * ctx_size; + ms_dump(M_HID_SETUP, "output_ctx_post_set_config", dev_ctx_phys, + (*dev_ctx).0.as_ptr(), out_ctx_len); } } - // Phase 2b: SET_INTERFACE + SET_PROTOCOL REMOVED. - // - // Linux ftrace confirmed: Linux does NOT send SET_INTERFACE or SET_PROTOCOL - // for HID devices on this Parallels vxHC. Per xHCI spec section 4.6.6, - // SET_INTERFACE requires the host to Deconfigure (DC=1) then re-Configure - // endpoints. Sending SET_INTERFACE as a raw USB control transfer without - // the xHCI-side deconfigure/reconfigure may cause the Parallels vxHC to - // internally mark endpoints as needing reconfiguration, leading to CC=12. - // ========================================================================= - // Phase 3: HID interface setup (SET_IDLE, GET_REPORT_DESC, etc.) + // Phase 2+3 (merged): Per-interface atomic setup — matches Linux ordering. + // + // Linux does all setup for interface N before moving to interface N+1. + // Previously Phase 2 (SET_INTERFACE loop) and Phase 3 (per-interface class + // setup) were separate loops. Merging them ensures the vxHC sees the same + // per-interface atomic ordering as the Linux kernel module. + // + // Per-interface sequence: + // 1. SET_INTERFACE(alt=0) + // 2. SET_PROTOCOL(boot=0) — only if is_boot (subclass=1) + // 3. SET_IDLE + // 4. GET HID Report Descriptor + // 5. GET_REPORT/SET_REPORT (Feature) — mouse only + // 6. SET_REPORT (LED) — keyboard only // ========================================================================= + ms_kv!(M_HID_SETUP, "hid_interfaces={} slot={}", iface_count, slot_id); + for i in 0..iface_count { if let Some(ref info) = ifaces[i] { + // Step 1: SET_INTERFACE(alt=0) — matches working Linux module. + ms_kv!(M_HID_SETUP, "SET_INTERFACE slot={} iface={}", slot_id, info.interface_number); + match set_interface(state, slot_id, info.interface_number, 0) { + Ok(()) => {} + Err(_) => {} // Non-fatal: some devices may STALL + } + + if !MINIMAL_INIT { + // Step 2: SET_PROTOCOL(boot=0) for boot-class interfaces (subclass=1). + if info.is_boot { + ms_kv!(M_HID_SETUP, "SET_PROTOCOL slot={} iface={}", slot_id, info.interface_number); + match set_protocol(state, slot_id, info.interface_number, 0) { + Ok(()) => {} + Err(_) => {} // Non-fatal: some devices may STALL + } + } + } if info.is_nkro { // NKRO keyboard: SET_IDLE + GET_HID_REPORT_DESC then ep2in TRB. - // Linux sends these for the NKRO interface before ep2in (ftrace lines 900, 911, 923). if !MINIMAL_INIT { + ms_kv!(M_HID_SETUP, "SET_IDLE slot={} iface={}", slot_id, info.interface_number); match set_idle(state, slot_id, info.interface_number) { Ok(()) => {} Err(_) => {} @@ -2659,15 +3117,14 @@ fn configure_hid( state.kbd_slot = slot_id; state.kbd_nkro_endpoint = info.dci; - // Queue interrupt TRB immediately (matching Linux: queue right after HID setup). - let _ = DIAG_FIRST_QUEUE_SOURCE.compare_exchange(0, 1, Ordering::AcqRel, Ordering::Relaxed); - let _ = queue_hid_transfer(state, info.hid_idx, slot_id, info.dci); + // TRB queueing deferred to post-enumeration in init() to prevent + // Transfer Events from being silently consumed by subsequent + // control_transfer() and wait_for_command() calls. } else if info.is_keyboard { - // Boot keyboard: SET_IDLE + GET_HID_REPORT_DESC + SET_REPORT(LED=0) + ep1in TRB. - // Linux ftrace (lines 827, 838, 851, 863): - // SET_IDLE (iface=0) → GET_HID_REPORT_DESC (58 bytes) → SET_REPORT(LED=0) → ep1in TRB + // Boot keyboard: SET_IDLE + GET_HID_REPORT_DESC + SET_REPORT(LED=0). if !MINIMAL_INIT { + ms_kv!(M_HID_SETUP, "SET_IDLE slot={} iface={}", slot_id, info.interface_number); match set_idle(state, slot_id, info.interface_number) { Ok(()) => {} Err(_) => {} @@ -2681,13 +3138,21 @@ fn configure_hid( state.kbd_slot = slot_id; state.kbd_endpoint = info.dci; - // Queue interrupt TRB immediately (matching Linux: queue right after HID setup). - let _ = DIAG_FIRST_QUEUE_SOURCE.compare_exchange(0, 1, Ordering::AcqRel, Ordering::Relaxed); - let _ = queue_hid_transfer(state, info.hid_idx, slot_id, info.dci); + // TRB queueing deferred to post-enumeration in start_hid_polling() + // to prevent Transfer Events from being silently consumed by + // subsequent control_transfer() and wait_for_command() calls. } else { - // Mouse: GET_REPORT(Feature) + SET_REPORT(Feature). + // Mouse: SET_IDLE + GET_HID_REPORT_DESC + GET_REPORT(Feature) + SET_REPORT(Feature). + // Linux sends SET_IDLE and fetches the HID report descriptor for ALL + // HID interfaces, including mouse — not just keyboards. if !MINIMAL_INIT { + ms_kv!(M_HID_SETUP, "SET_IDLE slot={} iface={} (mouse)", slot_id, info.interface_number); + match set_idle(state, slot_id, info.interface_number) { + Ok(()) => {} + Err(_) => {} + } + fetch_hid_report_descriptor(state, slot_id, info.interface_number, info.hid_report_len); let feature_id: u8 = if info.hid_idx == 3 { 0x12 } else { 0x11 }; match get_set_feature_report(state, slot_id, info.interface_number, feature_id) { Ok(()) => {} @@ -2701,13 +3166,28 @@ fn configure_hid( state.mouse_endpoint = info.dci; } - // Queue interrupt TRB immediately (matching Linux: queue right after HID setup). - let _ = DIAG_FIRST_QUEUE_SOURCE.compare_exchange(0, 1, Ordering::AcqRel, Ordering::Relaxed); - let _ = queue_hid_transfer(state, info.hid_idx, slot_id, info.dci); + // TRB queueing deferred to post-enumeration in start_hid_polling() + // to prevent Transfer Events from being silently consumed by + // subsequent control_transfer() and wait_for_command() calls. } } } + // Dump output device context after all per-interface HID setup + if ep_count > 0 && max_dci != 0 { + let slot_idx = (slot_id - 1) as usize; + let ctx_size = state.context_size; + unsafe { + let dev_ctx = &raw const DEVICE_CONTEXTS[slot_idx]; + dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); + let dev_ctx_phys = virt_to_phys(&raw const DEVICE_CONTEXTS[slot_idx] as u64); + let out_ctx_len = (1 + max_dci as usize) * ctx_size; + ms_dump(M_HID_SETUP, "output_ctx_post_hid_setup", dev_ctx_phys, + (*dev_ctx).0.as_ptr(), out_ctx_len); + } + } + + ms_pass!(M_HID_SETUP); Ok(()) } @@ -2720,14 +3200,15 @@ fn queue_hid_transfer( ) -> Result<(), &'static str> { let ring_idx = HID_RING_BASE + hid_idx; - // Determine the physical address and size of the report buffer. - // Linux ftrace confirms: xhci_urb_enqueue lengths are the actual data sizes - // (8 bytes for kbd boot, 8 bytes for mouse, 9 bytes for NKRO/mouse2), NOT maxp. + // Linux uses the endpoint's maxPacketSize (64) as the TRB transfer length, + // not the HID report size (8 or 9). The xHC may write up to maxPacketSize + // bytes and ISP (Interrupt on Short Packet) handles the common case where + // the device sends fewer bytes than the buffer size. let (buf_phys, buf_len) = match hid_idx { - 0 => (virt_to_phys((&raw const KBD_REPORT_BUF) as u64), 8usize), - 2 => (virt_to_phys((&raw const NKRO_REPORT_BUF) as u64), 9usize), - 3 => (virt_to_phys((&raw const MOUSE2_REPORT_BUF) as u64), 9usize), - _ => (virt_to_phys((&raw const MOUSE_REPORT_BUF) as u64), 8usize), + 0 => (virt_to_phys((&raw const KBD_REPORT_BUF) as u64), 64usize), + 2 => (virt_to_phys((&raw const NKRO_REPORT_BUF) as u64), 64usize), + 3 => (virt_to_phys((&raw const MOUSE2_REPORT_BUF) as u64), 64usize), + _ => (virt_to_phys((&raw const MOUSE_REPORT_BUF) as u64), 64usize), }; // Fill report buffer with sentinel (0xDE) before giving it to the controller. @@ -2827,13 +3308,15 @@ fn queue_hid_transfer( Ok(()) } -/// Synchronously poll the event ring for a Transfer Event. -/// Returns the completion code (0xFF = timeout after ~500ms). -/// Traces the Transfer Event TRB for diagnostic purposes. -fn sync_poll_transfer_event(state: &XhciState) -> u32 { - let mut cc: u32 = 0xFF; - for _attempt in 0..500_000u32 { - unsafe { +/// Drain any stale events left in the event ring after enumeration. +/// +/// During enumeration, some xHCI controllers may leave Transfer Events +/// (e.g., Short Packet events for Data Stage TRBs) that weren't consumed +/// by wait_for_event. These must be drained before starting HID polling. +fn drain_stale_events(state: &XhciState) { + let mut drained = 0u32; + unsafe { + loop { let ring = &raw const EVENT_RING; let idx = EVENT_RING_DEQUEUE; let cycle = EVENT_RING_CYCLE; @@ -2846,71 +3329,8 @@ fn sync_poll_transfer_event(state: &XhciState) -> u32 { let trb = core::ptr::read_volatile(&(*ring).0[idx]); let trb_cycle = trb.control & 1 != 0; - if trb_cycle == cycle { - let trb_type_val = trb.trb_type(); - if trb_type_val == trb_type::TRANSFER_EVENT { - cc = trb.completion_code(); - // Trace the full Transfer Event TRB (shows slot/dci/pointer) - xhci_trace_trb(XhciTraceOp::TransferEvent, 0, 0, &trb); - let _ = DIAG_FIRST_XFER_CC.compare_exchange( - 0xFF, cc, Ordering::AcqRel, Ordering::Relaxed, - ); - let _ = DIAG_FIRST_XFER_PTR.compare_exchange( - 0, trb.param, Ordering::AcqRel, Ordering::Relaxed, - ); - } - // Advance dequeue for any event type - EVENT_RING_DEQUEUE = (idx + 1) % EVENT_RING_SIZE; - if EVENT_RING_DEQUEUE == 0 { - EVENT_RING_CYCLE = !cycle; - } - let ir0 = state.rt_base + 0x20; - let erdp_phys = virt_to_phys(&raw const EVENT_RING as u64) - + (EVENT_RING_DEQUEUE as u64) * 16; - write64(ir0 + 0x18, erdp_phys | (1 << 3)); - // Clear IMAN.IP + USBSTS.EINT (match C harness) - let iman = read32(ir0); - if iman & 1 != 0 { - write32(ir0, iman | 1); - } - let usbsts = read32(state.op_base + 0x04); - if usbsts & (1 << 3) != 0 { - write32(state.op_base + 0x04, usbsts | (1 << 3)); - } - if trb_type_val == trb_type::TRANSFER_EVENT { - break; - } - // Non-transfer event: continue polling - } - } - core::hint::spin_loop(); - } - cc -} - -/// Drain any stale events left in the event ring after enumeration. -/// -/// During enumeration, some xHCI controllers may leave Transfer Events -/// (e.g., Short Packet events for Data Stage TRBs) that weren't consumed -/// by wait_for_event. These must be drained before starting HID polling. -fn drain_stale_events(state: &XhciState) { - let mut drained = 0u32; - unsafe { - loop { - let ring = &raw const EVENT_RING; - let idx = EVENT_RING_DEQUEUE; - let cycle = EVENT_RING_CYCLE; - - dma_cache_invalidate( - &(*ring).0[idx] as *const Trb as *const u8, - core::mem::size_of::(), - ); - - let trb = core::ptr::read_volatile(&(*ring).0[idx]); - let trb_cycle = trb.control & 1 != 0; - - if trb_cycle != cycle { - break; // No more events + if trb_cycle != cycle { + break; // No more events } @@ -3251,18 +3671,230 @@ fn reset_halted_endpoint( /// Post-enumeration setup: drain stale events, mark HID polling as active. /// -/// Interrupt TRBs are queued post-init (after XHCI_INITIALIZED). This function -/// only drains stale events and marks polling as active. +/// Called after all device enumeration is complete. Drains stale events, then +/// queues interrupt TRBs for all configured HID endpoints and rings doorbells. +/// +/// TRBs are queued HERE instead of inline during Phase 3 of configure_hid() +/// because control_transfer() and wait_for_event_inner() silently consume +/// Transfer Events for non-EP0 endpoints. Queueing inline meant the Transfer +/// Events from earlier interfaces' TRBs were eaten by subsequent control +/// transfers for later interfaces, leaving no pending TRBs after init. fn start_hid_polling(state: &XhciState) { + // --- M10: INTERRUPT_TRANSFER --- + ms_begin!(M_INTR_XFER); + // Drain any stale Transfer Events that may have been generated during // port scanning or previous enumeration attempts. drain_stale_events(state); - // TRBs are now queued inline during enumeration (matching Linux's flow: - // queue immediately after each interface's HID setup). Set flags so the - // deferred path and heartbeat know TRBs have been queued. + // Set flags so the poll path and heartbeat know TRBs have been queued. KBD_TRB_FIRST_QUEUED.store(true, Ordering::Release); HID_TRBS_QUEUED.store(true, Ordering::Release); + let _ = DIAG_FIRST_QUEUE_SOURCE.compare_exchange(0, 1, Ordering::AcqRel, Ordering::Relaxed); + + // Queue interrupt TRBs for all configured HID endpoints. + // Keyboard boot + if state.kbd_slot != 0 && state.kbd_endpoint != 0 { + ms_kv!(M_INTR_XFER, "queued+doorbell: slot={} dci={} hid_idx=0", state.kbd_slot, state.kbd_endpoint); + let _ = queue_hid_transfer(state, 0, state.kbd_slot, state.kbd_endpoint); + } + // Keyboard NKRO + if state.kbd_slot != 0 && state.kbd_nkro_endpoint != 0 { + ms_kv!(M_INTR_XFER, "queued+doorbell: slot={} dci={} hid_idx=2", state.kbd_slot, state.kbd_nkro_endpoint); + let _ = queue_hid_transfer(state, 2, state.kbd_slot, state.kbd_nkro_endpoint); + } + // Mouse boot + if state.mouse_slot != 0 && state.mouse_endpoint != 0 { + ms_kv!(M_INTR_XFER, "queued+doorbell: slot={} dci={} hid_idx=1", state.mouse_slot, state.mouse_endpoint); + let _ = queue_hid_transfer(state, 1, state.mouse_slot, state.mouse_endpoint); + } + // Mouse2 + if state.mouse_slot != 0 && state.mouse_nkro_endpoint != 0 { + ms_kv!(M_INTR_XFER, "queued+doorbell: slot={} dci={} hid_idx=3", state.mouse_slot, state.mouse_nkro_endpoint); + let _ = queue_hid_transfer(state, 3, state.mouse_slot, state.mouse_nkro_endpoint); + } + + // Stop MMIO write tracing (dump disabled — CC=12 investigation complete) + MMIO_TRACE_ACTIVE.store(false, Ordering::Release); + + ms_pass!(M_INTR_XFER); +} + +// ============================================================================= +// INHERIT_UEFI: Endpoint Discovery and Redirection +// ============================================================================= + +/// Inherit UEFI's configured endpoints instead of re-enumerating from scratch. +/// +/// Walks our DEVICE_CONTEXTS (which have UEFI's data copied in), discovers +/// interrupt IN endpoints, assigns them to xhci_state fields, and redirects +/// each endpoint's transfer ring to our memory via StopEndpoint + SetTRDequeuePointer. +fn inherit_uefi_endpoints(state: &mut XhciState) -> Result<(), &'static str> { + let context_size = state.context_size; + let mut discovered_slots: [(u8, u8, u8); 8] = [(0, 0, 0); 8]; // (slot_id, dci_boot, dci_nkro) + let mut num_discovered = 0usize; + + // Walk all slots and find those with interrupt IN endpoints + for slot_id in 1..=state.max_slots { + let slot_idx = (slot_id as usize) - 1; + let dcbaa_entry = unsafe { + let dcbaa = &raw const DCBAA; + (*dcbaa).0[slot_id as usize] + }; + if dcbaa_entry == 0 { + continue; + } + + unsafe { + let dev_ctx = &raw const DEVICE_CONTEXTS[slot_idx]; + dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); + + // Slot Context DW0: bits 31:27 = Context Entries (num DCIs with valid data) + let slot_dw0 = core::ptr::read_volatile((*dev_ctx).0.as_ptr() as *const u32); + let num_entries = (slot_dw0 >> 27) & 0x1F; + + // Slot Context DW3: bits 31:27 = Slot State + let slot_dw3 = core::ptr::read_volatile( + (*dev_ctx).0.as_ptr().add(12) as *const u32, + ); + let slot_state = (slot_dw3 >> 27) & 0x1F; + + // Slot states: 0=Disabled/Enabled, 1=Default, 2=Addressed, 3=Configured + if slot_state < 2 { + continue; + } + + let mut boot_dci: u8 = 0; + let mut nkro_dci: u8 = 0; + + for dci in 1..=num_entries { + let ep_base = (*dev_ctx).0.as_ptr().add(dci as usize * context_size); + let ep_dw0 = core::ptr::read_volatile(ep_base as *const u32); + let ep_dw1 = core::ptr::read_volatile(ep_base.add(4) as *const u32); + let ep_state = ep_dw0 & 0x7; + let ep_type = (ep_dw1 >> 3) & 0x7; + // EP Type 7 = Interrupt IN + if ep_type == 7 && ep_state != 0 { + if boot_dci == 0 { + boot_dci = dci as u8; + } else if nkro_dci == 0 { + nkro_dci = dci as u8; + } + } + } + + if boot_dci != 0 && num_discovered < 8 { + discovered_slots[num_discovered] = (slot_id, boot_dci, nkro_dci); + num_discovered += 1; + } + } + } + + if num_discovered == 0 { + return Err("INHERIT_UEFI: no interrupt IN endpoints found"); + } + + // Assign discovered slots to mouse/keyboard. + // Parallels typically: slot 1 = mouse, slot 2 = keyboard. + // We assign first discovered to mouse, second to keyboard. + if num_discovered >= 1 { + let (slot_id, boot_dci, nkro_dci) = discovered_slots[0]; + state.mouse_slot = slot_id; + state.mouse_endpoint = boot_dci; + state.mouse_nkro_endpoint = nkro_dci; + } + if num_discovered >= 2 { + let (slot_id, boot_dci, nkro_dci) = discovered_slots[1]; + state.kbd_slot = slot_id; + state.kbd_endpoint = boot_dci; + state.kbd_nkro_endpoint = nkro_dci; + } + + // For each discovered endpoint, redirect its transfer ring to ours. + // Sequence: StopEndpoint (if Running) → SetTRDequeuePointer → ready for Normal TRBs. + let endpoints_to_redirect: [(u8, u8, usize); 4] = [ + // (slot_id, dci, hid_idx) + (state.mouse_slot, state.mouse_endpoint, 1), // hid_idx 1 = mouse boot + (state.mouse_slot, state.mouse_nkro_endpoint, 3), // hid_idx 3 = mouse2 + (state.kbd_slot, state.kbd_endpoint, 0), // hid_idx 0 = kbd boot + (state.kbd_slot, state.kbd_nkro_endpoint, 2), // hid_idx 2 = kbd NKRO + ]; + + for &(slot_id, dci, hid_idx) in &endpoints_to_redirect { + if slot_id == 0 || dci == 0 { + continue; + } + + // Read current endpoint state from our copy of the output context + let slot_idx = (slot_id as usize) - 1; + let ep_state = unsafe { + let dev_ctx = &raw const DEVICE_CONTEXTS[slot_idx]; + dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); + let ep_base = (*dev_ctx).0.as_ptr().add(dci as usize * context_size); + core::ptr::read_volatile(ep_base as *const u32) & 0x7 + }; + + // StopEndpoint if Running (state=1) or Halted (state=2) + // State 0=Disabled, 1=Running, 2=Halted, 3=Stopped + if ep_state == 1 || ep_state == 2 { + if ep_state == 2 { + // Halted → ResetEndpoint first to get to Stopped + let reset_trb = Trb { + param: 0, + status: 0, + control: (trb_type::RESET_ENDPOINT << 10) + | ((slot_id as u32) << 24) + | ((dci as u32) << 16), + }; + enqueue_command(reset_trb); + ring_doorbell(state, 0, 0); + let _ = wait_for_command(state); + } else { + // Running → StopEndpoint to get to Stopped + let stop_trb = Trb { + param: 0, + status: 0, + control: (trb_type::STOP_ENDPOINT << 10) + | ((slot_id as u32) << 24) + | ((dci as u32) << 16), + }; + enqueue_command(stop_trb); + ring_doorbell(state, 0, 0); + let _ = wait_for_command(state); + } + } + + // SetTRDequeuePointer: redirect to our transfer ring + let ring_idx = HID_RING_BASE + hid_idx; + // Zero the transfer ring and reset enqueue index + unsafe { + core::ptr::write_bytes( + TRANSFER_RINGS[ring_idx].as_mut_ptr(), + 0, + TRANSFER_RING_SIZE, + ); + TRANSFER_ENQUEUE[ring_idx] = 0; + TRANSFER_CYCLE[ring_idx] = true; + dma_cache_clean( + TRANSFER_RINGS[ring_idx].as_ptr() as *const u8, + TRANSFER_RING_SIZE * core::mem::size_of::(), + ); + } + + let ring_phys = virt_to_phys(unsafe { (&raw const TRANSFER_RINGS[ring_idx]) as u64 }); + // DCS bit 0 = 1 (matches initial cycle state) + let set_deq_trb = Trb { + param: ring_phys | 1, + status: 0, + control: (trb_type::SET_TR_DEQUEUE_POINTER << 10) + | ((slot_id as u32) << 24) + | ((dci as u32) << 16), + }; + enqueue_command(set_deq_trb); + ring_doorbell(state, 0, 0); + let _ = wait_for_command(state); + } + Ok(()) } // ============================================================================= @@ -3272,14 +3904,14 @@ fn start_hid_polling(state: &XhciState) { /// Scan all root hub ports for connected devices, enumerate, and configure HID devices. fn scan_ports(state: &mut XhciState) -> Result<(), &'static str> { - // Dump PORTSC of all ports (especially USB 2.0 ports 12-13) - for port in 0..state.max_ports as u64 { - let portsc_addr = state.op_base + 0x400 + port * 0x10; - let portsc = read32(portsc_addr); - let ccs = portsc & 1; - if ccs != 0 || port >= 12 { - } - } + // CRITICAL: Clear ALL PORTSC change bits BEFORE any Enable Slot command. + // + // MMIO comparison shows Linux's hub driver clears CSC (Connection Status + // Change, bit 17) before sending Enable Slot. Breenix was sending Enable + // Slot with CSC still set. The Parallels virtual xHC may use CSC state to + // track whether the driver properly completed the port status change + // handshake before claiming the device. + let _early_cleared = clear_all_port_changes(state); let mut slots_used: u8 = 0; // MOUSE_ONLY: enumerate only 1 device (mouse on port 0), skip keyboard/composite. @@ -3302,17 +3934,38 @@ fn scan_ports(state: &mut XhciState) -> Result<(), &'static str> { continue; } + let port_id = port as u8 + 1; // 1-based port ID + // --- M5: PORT_DETECTION --- + ms_begin!(M_PORT_DET); + ms_kv!(M_PORT_DET, "port={} PORTSC=0x{:08x} CCS={} PED={}", + port_id, portsc, portsc & 1, (portsc >> 1) & 1); + ms_kv!(M_PORT_DET, "port={} speed_raw={} PRC={}", + port_id, (portsc >> 10) & 0xF, (portsc >> 21) & 1); - // Check if port is enabled (PED, bit 1) + // Port reset: only when PED=0 (matching Linux kernel module behavior). + // + // After HCRST, SuperSpeed ports auto-enable (PED=1). The Linux kernel + // module skips port reset when PED=1, and the Parallels hypervisor + // handles this correctly — AddressDevice internally triggers a device + // reset in the hypervisor's USB device model. + // + // Explicit port reset when PED=1 was previously added but may confuse + // the hypervisor's internal state machine. if portsc & (1 << 1) == 0 { - // Port not enabled - perform a port reset + ms_kv!(M_PORT_DET, "port={} resetting (PED=0)", port_id); // Write PR (Port Reset, bit 4). - // Note: PORTSC is a mix of RW, RW1C, and RO bits. We must preserve - // RW bits and NOT accidentally clear RW1C bits by writing 1 to them. - // RW1C bits in PORTSC: CSC(17), PEC(18), WRC(19), OCC(20), PRC(21), PLC(22), CEC(23) + // + // PORTSC bit types (xHCI spec Table 5-27): + // PED (bit 1): W1CS — writing 1 DISABLES the port! Must write 0. + // PR (bit 4): RW1S — writing 1 starts port reset + // Bits 17-23: RW1C — change status bits, writing 1 clears them + // + // All W1C/W1CS bits must be written as 0 to avoid side effects. + // This matches Linux's xhci_port_state_to_neutral(). let preserve_mask: u32 = !( + (1 << 1) | // PED - W1CS, writing 1 disables port! (1 << 17) | (1 << 18) | (1 << 19) | (1 << 20) | (1 << 21) | (1 << 22) | (1 << 23) ); write32(portsc_addr, (portsc & preserve_mask) | (1 << 4)); @@ -3324,37 +3977,58 @@ fn scan_ports(state: &mut XhciState) -> Result<(), &'static str> { ) .is_err() { + ms_kv!(M_PORT_DET, "port={} reset timeout", port_id); continue; } - // Clear PRC (W1C) and check that port is now enabled + // Clear PRC (W1C) — write 1 to bit 21 to clear it. + // Use the same preserve_mask to avoid disabling PED. let portsc_after = read32(portsc_addr); write32(portsc_addr, (portsc_after & preserve_mask) | (1 << 21)); let portsc_final = read32(portsc_addr); + ms_kv!(M_PORT_DET, "port={} post_reset PORTSC=0x{:08x} PED={}", + port_id, portsc_final, (portsc_final >> 1) & 1); if portsc_final & (1 << 1) == 0 { continue; } - + } else { + ms_kv!(M_PORT_DET, "port={} PED=1, skipping port reset (matching Linux module)", port_id); } + ms_pass!(M_PORT_DET); + + // --- M6: SLOT_ENABLE --- + ms_begin!(M_SLOT_EN); // Enable Slot for this device let slot_id = match enable_slot(state) { Ok(id) => id, Err(_) => { + ms_fail!(M_SLOT_EN, "EnableSlot returned error"); + ms_kv!(M_SLOT_EN, "port={} slot_id=0", port_id); continue; } }; if slot_id == 0 { + ms_fail!(M_SLOT_EN, "EnableSlot returned 0 or out of range"); + ms_kv!(M_SLOT_EN, "port={} slot_id=0", port_id); continue; } + ms_kv!(M_SLOT_EN, "port={} slot_id={}", port_id, slot_id); + ms_pass!(M_SLOT_EN); slots_used += 1; + // --- M7: DEVICE_ADDRESS --- + ms_kv!(M_ADDR_DEV, "port={} speed={} slot={}", + port_id, (read32(portsc_addr) >> 10) & 0xF, slot_id); + ms_begin!(M_ADDR_DEV); // Address Device (port numbers are 1-based) - if let Err(_) = address_device(state, slot_id, port as u8 + 1) { + if let Err(_) = address_device(state, slot_id, port_id) { + ms_fail!(M_ADDR_DEV, "command failed"); continue; } + ms_pass!(M_ADDR_DEV); // Linux USB 3.0 enumeration sequence (from ftrace): // 1. GET_DESCRIPTOR(Device, 8) — first 8 bytes for maxpkt0 @@ -3370,9 +4044,9 @@ fn scan_ports(state: &mut XhciState) -> Result<(), &'static str> { continue; } - // Step 2: SET_ISOCH_DELAY (between the two descriptor reads) - if let Err(_) = set_isoch_delay(state, slot_id) { - } + // Step 2: SET_ISOCH_DELAY — SKIPPED. + // Parallels' virtual xHCI STALLs this request, causing an EP0 reset. + // The working Linux module also does NOT send SET_ISOCH_DELAY. // Step 3: Full device descriptor (18 bytes) let mut desc_buf = [0u8; 18]; @@ -3506,13 +4180,84 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { XHCI_TRACE_ACTIVE.store(true, Ordering::Relaxed); xhci_trace_note(0, "init_start"); - // 1. Enable bus mastering + memory space - pci_dev.enable_bus_master(); + // DMA structures are now in .bss (WB-cacheable, zeroed by boot.S). + // No dc ivac needed — there are no stale NC cache lines to worry about. + + // Enable MemSpace + BusMaster immediately to keep the xHCI BAR mapped. + // UEFI firmware disables MemSpace during ExitBootServices cleanup, which + // triggers phymemrange_disable. We re-enable it right away to minimize + // the BAR-unmapped gap (matching linux-probe's ~95ms gap). + { + let cmd_before = crate::drivers::pci::pci_read_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, + ); + let cmd_new = cmd_before | 0x0006; // MemSpace + BusMaster + if cmd_new != cmd_before { + crate::drivers::pci::pci_write_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, cmd_new, + ); + } + } + + // Wait for Parallels' ASYNC OnExitBootServices handler to complete. + // + // CRITICAL FINDING: The hypervisor's OnExitBootServices() handler runs + // ASYNCHRONOUSLY ~440ms after the EBS call, and triggers a second XHC + // controller reset at ~685ms. If our HCRST happens BEFORE this async reset, + // the async reset destroys all endpoints we just created → CC=12. + // + // By waiting 1000ms, we ensure the async handler's XHC reset has already + // completed before we do our HCRST. Our HCRST becomes the LAST reset, + // and endpoint state created after it will persist. + // + // On linux-probe, ~1.8 seconds pass between EBS and the linux module's + // HCRST — well past the async handler window. + const EBS_SETTLE_MS: u32 = 1000; + delay_ms(EBS_SETTLE_MS); + + // Step 1: pci_enable_device() equivalent + // 1a. Transition to D0 power state (pci_set_power_state → pci_raw_set_power_state) + let _ = pci_dev.set_power_state_d0(); + // 1b. Enable Memory Space only (pci_enable_resources sets bit 1) + // Linux does NOT set Bus Master or INTx Disable here. pci_dev.enable_memory_space(); + // 1c. Clear Status register error bits (w1c) + { + let status = crate::drivers::pci::pci_read_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x06, + ); + if status & 0xF900 != 0 { + crate::drivers::pci::pci_write_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x06, status, + ); + } + } + + // Step 2: pci_set_master() equivalent — separate write for Bus Master + pci_dev.enable_bus_master(); + + // Note: Linux does NOT write Cache Line Size or Latency Timer for PCIe devices. + // The firmware defaults are preserved. We no longer write these either. + + // Verify the result + let pci_cmd_after = crate::drivers::pci::pci_read_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, + ); + if pci_cmd_after & 0x06 != 0x06 { + crate::serial_println!("[xhci-pci] WARNING: bus master or mem space NOT set!"); + } // 2. Map BAR0 via HHDM - let bar = pci_dev.get_mmio_bar().ok_or("XHCI: no MMIO BAR found")?; - let base = HHDM_BASE + bar.address; + // Read BAR0 directly from PCI config (not cached self.bars) because + // Step 0 may have reassigned BAR0 to a new address. + let bar0_raw = crate::drivers::pci::pci_read_config_dword( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x10, + ); + let bar0_phys = (bar0_raw & 0xFFFFF000) as u64; + let base = HHDM_BASE + bar0_phys; + + // Store BAR base for MMIO write tracing (write32 computes offset = addr - base) + MMIO_TRACE_BAR_BASE.store(base, Ordering::Relaxed); // 3. Read capability registers let cap_word = read32(base); @@ -3538,6 +4283,15 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { let rt_base = base + rts_offset as u64; let db_base = base + db_offset as u64; + // --- M1: CONTROLLER_DISCOVERY --- + ms_begin!(M_DISCOVERY); + ms_kv!(M_DISCOVERY, "BAR0_phys=0x{:x}", bar0_phys); + ms_kv!(M_DISCOVERY, "xHCI_version=0x{:04x}", (cap_word >> 16) & 0xFFFF); + ms_kv!(M_DISCOVERY, "max_slots={} max_ports={} ctx_size={}", max_slots, max_ports, context_size); + ms_kv!(M_DISCOVERY, "cap_len={} op_off=0x{:x} rt_off=0x{:x} db_off=0x{:x}", + cap_length, cap_length as u32, rts_offset, db_offset); + ms_kv!(M_DISCOVERY, "HCSPARAMS1=0x{:08x} HCCPARAMS1=0x{:08x}", hcsparams1, hccparams1); + ms_pass!(M_DISCOVERY); // 3b. Walk Extended Capabilities list for Supported Protocol info. // HCCPARAMS1 bits 31:16 = xECP (xHCI Extended Capabilities Pointer) in DWORDs from base. @@ -3549,7 +4303,34 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { let cap_id = ecap_dw0 & 0xFF; let next_ptr = (ecap_dw0 >> 8) & 0xFF; - if cap_id == 2 { + if cap_id == 1 { + // USB Legacy Support Capability (USBLEGSUP, ID=1) + // xHCI spec 7.1.1: BIOS/OS handoff to claim controller from UEFI. + // DW0 bits: [16] HC BIOS Owned Semaphore, [24] HC OS Owned Semaphore + let bios_owned = (ecap_dw0 >> 16) & 1; + if bios_owned != 0 { + // Set OS Owned Semaphore (bit 24), wait for BIOS to release (bit 16 clears) + write32(ecap_addr, ecap_dw0 | (1 << 24)); + for _ in 0..1_000_000u32 { + let v = read32(ecap_addr); + if (v >> 16) & 1 == 0 { + break; + } + core::hint::spin_loop(); + } + xhci_trace_note(0, "usblegsup_claimed"); + } else { + // BIOS doesn't own it, just set OS owned + write32(ecap_addr, ecap_dw0 | (1 << 24)); + xhci_trace_note(0, "usblegsup_no_bios"); + } + // Also disable SMI on USB events (USBLEGCTLSTS at ecap_addr + 4) + // Set bits 1, 4, 13, 14, 15 to disable SMI routing + let legctl = read32(ecap_addr + 4); + write32(ecap_addr + 4, legctl & !( + (1 << 0) | (1 << 1) | (1 << 4) | (1 << 13) | (1 << 14) | (1 << 15) + )); + } else if cap_id == 2 { // Supported Protocol Capability (ID=2) // DW0: cap_id(7:0), next(15:8), minor_rev(23:16), major_rev(31:24) // DW1: Name String (ASCII, e.g., "USB ") @@ -3571,92 +4352,97 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { } else { } - // 3c. EHCI BIOS handoff — claim the companion EHCI controller. + // 3.5. PCI Bus Master disable→enable cycle // - // Parallels exposes Intel 82801FB EHCI (0x8086:0x265c) alongside the NEC xHC. - // Internal Parallels logs show EHC controller resets ~30ms after our - // ConfigureEndpoint commands, followed by "DisableEndpoint while io_cnt is - // not zero!" — which kills interrupt endpoints and causes CC=12. + // On Linux, when xhci_hcd is unbound and our module binds, the PCI device goes through: + // pci_disable_device() → clears Bus Master (CMD bit 2) + // pci_enable_device() → sets Memory Space (CMD bit 1) + // pci_set_master() → sets Bus Master (CMD bit 2) // - // By claiming OS ownership of EHCI (USBLEGSUP handoff) and halting it BEFORE - // our HCRST, we may prevent Parallels' internal EHC reset cascade. - if let Some(ehci_dev) = crate::drivers::pci::find_device(0x8086, 0x265c) { - xhci_trace_note(0, "ehci_claim"); - - // EHCI USBLEGSUP is at PCI config offset 0x60 for Intel controllers. - // Bits: [7:0]=cap_id(0x01), [15:8]=next_cap, [16]=BIOS_owned, [24]=OS_owned - let usblegsup = crate::drivers::pci::pci_read_config_dword( - ehci_dev.bus, ehci_dev.device, ehci_dev.function, 0x60); - - if usblegsup & 0xFF == 0x01 { - // Valid USBLEGSUP capability — claim OS ownership - crate::drivers::pci::pci_write_config_dword( - ehci_dev.bus, ehci_dev.device, ehci_dev.function, 0x60, - usblegsup | (1 << 24)); - - // Wait for BIOS Owned (bit 16) to clear — up to 100ms - for _ in 0..100_000u32 { - let val = crate::drivers::pci::pci_read_config_dword( - ehci_dev.bus, ehci_dev.device, ehci_dev.function, 0x60); - if val & (1 << 16) == 0 { - break; - } - core::hint::spin_loop(); - } - xhci_trace_note(0, "ehci_legsup"); - } - - // Map EHCI BAR0 and halt the controller - if let Some(ehci_bar) = ehci_dev.get_mmio_bar() { - let ehci_base = HHDM_BASE + ehci_bar.address; - let ehci_cap_length = read32(ehci_base) & 0xFF; - let ehci_op_base = ehci_base + ehci_cap_length as u64; - - // USBCMD at op_base + 0x00: clear RS (bit 0) to halt - let ehci_cmd = read32(ehci_op_base); - write32(ehci_op_base, ehci_cmd & !1); - - // Wait for HCHalted (USBSTS bit 12) — up to ~10ms - for _ in 0..100_000u32 { - let sts = read32(ehci_op_base + 0x04); - if sts & (1 << 12) != 0 { - break; - } - core::hint::spin_loop(); - } + // This disable→enable cycle may signal the Parallels hypervisor to reinitialize + // its internal USB device model. Breenix never goes through this cycle because + // the UEFI firmware leaves Bus Master enabled and we just confirm it. + { + let cmd = crate::drivers::pci::pci_read_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, + ); - // Also write EHCI USBCMD = 0 to disable all functionality - write32(ehci_op_base, 0); - xhci_trace_note(0, "ehci_halted"); - } + // Disable Bus Master (clear bit 2), keep Memory Space (bit 1) + let cmd_no_bm = cmd & !0x0004; + crate::drivers::pci::pci_write_config_word( + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, cmd_no_bm, + ); + delay_ms(10); // Brief settle - // Disable EHCI bus mastering at PCI level to prevent any DMA activity - let cmd = crate::drivers::pci::pci_read_config_word( - ehci_dev.bus, ehci_dev.device, ehci_dev.function, 0x04); + // Re-enable Bus Master crate::drivers::pci::pci_write_config_word( - ehci_dev.bus, ehci_dev.device, ehci_dev.function, 0x04, - cmd & !0x06); // Clear bus master (bit 2) + memory space (bit 1) - xhci_trace_note(0, "ehci_pci_off"); + pci_dev.bus, pci_dev.device, pci_dev.function, 0x04, cmd, + ); + delay_ms(10); + } - // 4. Stop controller: clear USBCMD.RS, wait for USBSTS.HCH + // 4. Stop controller + HCRST + + // Enable MMIO write tracing: capture every write32() from halt/HCRST onward + MMIO_TRACE_IDX.store(0, Ordering::Relaxed); + MMIO_TRACE_ACTIVE.store(true, Ordering::Release); + + // 4a. Halt the controller (RS=0) let usbcmd = read32(op_base); if usbcmd & 1 != 0 { - // Controller is running, stop it write32(op_base, usbcmd & !1); wait_for(|| read32(op_base + 0x04) & 1 != 0, 100_000) .map_err(|_| "XHCI: timeout waiting for HCH")?; xhci_trace_note(0, "ctrl_stopped"); } - // 5. Reset: set USBCMD.HCRST, wait for clear - write32(op_base, read32(op_base) | 2); - wait_for(|| read32(op_base) & 2 == 0, 100_000) - .map_err(|_| "XHCI: timeout waiting for HCRST clear")?; - // Wait for CNR (Controller Not Ready, bit 11 of USBSTS) to clear - wait_for(|| read32(op_base + 0x04) & (1 << 11) == 0, 100_000) - .map_err(|_| "XHCI: timeout waiting for CNR clear")?; - xhci_trace_note(0, "ctrl_reset"); + if INHERIT_UEFI { + // INHERIT_UEFI: Skip HCRST. Read UEFI's device context state, then + // fall through to normal data structure setup (command ring, event ring). + // After the controller is running, we'll redirect UEFI's interrupt + // endpoints to our transfer rings instead of re-enumerating. + xhci_trace_note(0, "inherit_uefi"); + + // Read UEFI's DCBAAP before we overwrite it + let uefi_dcbaap = read64(op_base + 0x30); + unsafe { UEFI_DCBAAP_SAVED = uefi_dcbaap; } + } else if SKIP_HCRST { + xhci_trace_note(0, "ctrl_halt_no_reset"); + } else { + // Full HCRST + write32(op_base, read32(op_base) | (1 << 1)); + wait_for(|| read32(op_base) & (1 << 1) == 0, 500_000) + .map_err(|_| "XHCI: timeout waiting for HCRST to clear")?; + wait_for(|| read32(op_base + 0x04) & (1 << 11) == 0, 500_000) + .map_err(|_| "XHCI: timeout waiting for CNR to clear after reset")?; + xhci_trace_note(0, "ctrl_reset"); + + // EHCI reset DISABLED: The Parallels hypervisor's EHC reset triggers a + // CASCADING second XHC controller reset, which destroys the endpoint state + // that was just created after our first HCRST. Hypervisor log evidence: + // 45.571: "XHC controller reset" + "EHC controller reset" (same timestamp) + // 45.571: "DisableEndpoint while io_cnt is not zero!" (endpoints destroyed) + // After the cascading 2nd reset: NO ep creates → CC=12. + // On linux-probe, the linux module does NOT reset the EHCI controller. + } + + // --- M2: CONTROLLER_RESET --- + ms_begin!(M_RESET); + { + let usbsts_post = read32(op_base + 0x04); + ms_kv!(M_RESET, "USBSTS=0x{:08x} HCH={} CNR={}", usbsts_post, + usbsts_post & 1, (usbsts_post >> 11) & 1); + ms_regs(M_RESET, op_base, rt_base + 0x20); + if (usbsts_post & 1) != 0 && (usbsts_post & (1 << 11)) == 0 { + ms_pass!(M_RESET); + } else { + ms_fail!(M_RESET, "HCH or CNR unexpected"); + } + } + + // --- M3: DATA_STRUCTURES --- + ms_begin!(M_DATA_STRUC); // 6. Set MaxSlotsEn let slots_en = max_slots.min(MAX_SLOTS as u8); @@ -3674,7 +4460,55 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { core::ptr::write_bytes((*dcbaa).0.as_mut_ptr(), 0, 256); dma_cache_clean((*dcbaa).0.as_ptr() as *const u8, 256 * core::mem::size_of::()); } + // INHERIT_UEFI: copy UEFI's device context data into our DEVICE_CONTEXTS + // arrays BEFORE writing DCBAAP, so the xHC sees UEFI's endpoint state + // in our memory. + if INHERIT_UEFI { + let uefi_dcbaap = unsafe { UEFI_DCBAAP_SAVED }; + if uefi_dcbaap != 0 { + let uefi_dcbaa_virt = HHDM_BASE + uefi_dcbaap; + dma_cache_invalidate(uefi_dcbaa_virt as *const u8, 256 * 8); + for slot_id in 1..=slots_en { + let ctx_ptr_phys = unsafe { + core::ptr::read_volatile( + (uefi_dcbaa_virt + slot_id as u64 * 8) as *const u64, + ) + }; + if ctx_ptr_phys == 0 { + continue; + } + let slot_idx = (slot_id as usize) - 1; + let src_virt = HHDM_BASE + ctx_ptr_phys; + unsafe { + dma_cache_invalidate(src_virt as *const u8, 4096); + // Copy UEFI's output context into our DEVICE_CONTEXTS + core::ptr::copy_nonoverlapping( + src_virt as *const u8, + DEVICE_CONTEXTS[slot_idx].0.as_mut_ptr(), + 4096, + ); + dma_cache_clean(DEVICE_CONTEXTS[slot_idx].0.as_ptr(), 4096); + // Set DCBAA entry to point to our copy + let our_ctx_phys = + virt_to_phys((&raw const DEVICE_CONTEXTS[slot_idx]) as u64); + let dcbaa = &raw mut DCBAA; + (*dcbaa).0[slot_id as usize] = our_ctx_phys; + } + } + // Re-clean DCBAA after copying entries + unsafe { + dma_cache_clean( + (*(&raw const DCBAA)).0.as_ptr() as *const u8, + 256 * core::mem::size_of::(), + ); + } + } + } write64(op_base + 0x30, dcbaa_phys); + // Readback DCBAAP to verify write (volatile read ensures write posted) + let _dcbaap_rb = read64(op_base + 0x30); + ms_kv!(M_DATA_STRUC, "DCBAA: phys=0x{:x} written=0x{:x} readback=0x{:x}", + dcbaa_phys, dcbaa_phys, _dcbaap_rb); // 8. Set Command Ring Control Register (CRCR) let cmd_ring_phys = virt_to_phys((&raw const CMD_RING) as u64); @@ -3687,7 +4521,10 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { dma_cache_clean((*ring).0.as_ptr() as *const u8, CMD_RING_SIZE * core::mem::size_of::()); } // CRCR: physical address | RCS (Ring Cycle State) = 1 - write64(op_base + 0x18, cmd_ring_phys | 1); + let crcr_val = cmd_ring_phys | 1; + write64(op_base + 0x18, crcr_val); + ms_kv!(M_DATA_STRUC, "CMD_RING: phys=0x{:x} CRCR_written=0x{:x} readback=0x{:x}", + cmd_ring_phys, crcr_val, read64(op_base + 0x18)); // 9. Set up Event Ring for Interrupter 0 let event_ring_phys = virt_to_phys((&raw const EVENT_RING) as u64); @@ -3713,101 +4550,89 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { let ir0 = rt_base + 0x20; // Interrupter 0 register set + // Match Linux module register write order: IMOD before ERSTSZ/ERDP/ERSTBA. + // Linux's xhci_setup_rings() writes IMOD first (line 1155), then event ring regs. + write32(ir0 + 0x04, 0x000000a0); // IMOD = 160 * 250ns = 40µs + // ERSTSZ (Event Ring Segment Table Size) = 1 segment write32(ir0 + 0x08, 1); // ERDP (Event Ring Dequeue Pointer) = start of event ring write64(ir0 + 0x18, event_ring_phys); // ERSTBA (Event Ring Segment Table Base Address) - must be written AFTER ERSTSZ write64(ir0 + 0x10, erst_phys); - - - // 10. Configure MSI for event delivery (restored — hypothesis #24 disproved). - let irq = setup_xhci_msi(pci_dev); - XHCI_IRQ.store(irq, Ordering::Release); - - // 11. Enable interrupts on Interrupter 0 - // Set IMOD (Interrupt Moderation) — match Linux (0xa0 = 160 * 250ns = 40µs) - write32(ir0 + 0x04, 0x000000a0); + ms_kv!(M_DATA_STRUC, "EVT_RING: seg_phys=0x{:x}", event_ring_phys); + ms_kv!(M_DATA_STRUC, "ERST: phys=0x{:x} seg_addr=0x{:x} seg_size={}", + erst_phys, event_ring_phys, EVENT_RING_SIZE); + ms_kv!(M_DATA_STRUC, "ERDP: written=0x{:x} readback=0x{:x}", + event_ring_phys, read64(ir0 + 0x18)); + ms_kv!(M_DATA_STRUC, "ERSTBA: written=0x{:x} readback=0x{:x}", + erst_phys, read64(ir0 + 0x10)); + ms_regs(M_DATA_STRUC, op_base, ir0); + ms_pass!(M_DATA_STRUC); + + // 10. MSI is NOT configured here — matching Linux module order. + // The Linux module (breenix_xhci_probe.c) calls pci_alloc_irq_vectors() + // AFTER all enumeration, TRB queuing, and doorbell ringing (line 2229). + // During enumeration, Linux has: no MSI configured, INTx enabled. + // MSI will be configured after start_hid_polling() below. + + // 11. Enable interrupts on Interrupter 0 (needed for event ring operation). + // IMOD already written above (matching Linux's order: IMOD before event ring regs). let iman = read32(ir0); write32(ir0, iman | 2); // IMAN.IE = 1 // 12. Start controller: USBCMD.RS=1, INTE=1 + // INTE is needed for the controller to write events to the event ring. let usbcmd = read32(op_base); write32(op_base, usbcmd | 1 | (1 << 2)); // RS=1, INTE=1 - // Wait a bit for ports to detect connections - for _ in 0..100_000 { - core::hint::spin_loop(); + // --- M4: CONTROLLER_RUNNING --- + ms_begin!(M_RUNNING); + { + let cmd = read32(op_base); + let iman_val = read32(ir0); + if (cmd & 1) != 0 && (cmd & (1 << 2)) != 0 && (iman_val & 2) != 0 { + ms_pass!(M_RUNNING); + } else { + ms_fail!(M_RUNNING, "RS/INTE/IE not set"); + } } + // MATCH LINUX MODULE: enumerate IMMEDIATELY after RS=1. + // The working Linux module (breenix_xhci_probe.c) does NOT: + // - Send NEC vendor command (GET_FW type 49) + // - Clear PORTSC change bits before enumeration + // - Wait 2000ms before first enumeration + // It enumerates immediately, then msleep(2000), then enumerates again. + // + // Previously, Breenix had all of these extras and still got CC=12. + // Stripping them to match the working Linux module exactly. + + // Brief delay after RS=1 for periodic schedule to start + delay_ms(20); + // Verify controller is running let usbsts = read32(op_base + 0x04); if usbsts & 1 != 0 { xhci_trace_note(0, "err:ctrl_halted"); } - // 12b. NEC vendor command: GET_FW (matches Linux xhci_run() for NEC hosts). - // Linux sends TRB type 49 (NEC_GET_FW) immediately after RS=1 when vendor=0x1033. - // The Parallels vxHC emulates NEC uPD720200 and may use this as an init signal. - { - let nec_trb = Trb { - param: 0, - status: 0, - control: (trb_type::NEC_GET_FW << 10), - }; - xhci_trace_trb(XhciTraceOp::CommandSubmit, 0, 0, &nec_trb); - enqueue_command(nec_trb); - // Ring host controller doorbell (slot 0, target 0) — state not yet created. - write32(db_base, 0); - unsafe { core::arch::asm!("dsb sy", options(nostack, preserves_flags)); } - - // Poll for the vendor command completion (type 48 or standard type 33). - // Timeout after ~10ms — the NEC FW query is optional. - let mut got_response = false; - for _ in 0..100_000u32 { - unsafe { - let ring = &raw const EVENT_RING; - let idx = EVENT_RING_DEQUEUE; - let cycle = EVENT_RING_CYCLE; - dma_cache_invalidate( - &(*ring).0[idx] as *const Trb as *const u8, - core::mem::size_of::(), - ); - let trb = core::ptr::read_volatile(&(*ring).0[idx]); - let trb_cycle = trb.control & 1 != 0; - if trb_cycle == cycle { - let ttype = trb.trb_type(); - xhci_trace_trb(XhciTraceOp::CommandComplete, 0, 0, &trb); - // Advance dequeue - EVENT_RING_DEQUEUE = (idx + 1) % EVENT_RING_SIZE; - if EVENT_RING_DEQUEUE == 0 { EVENT_RING_CYCLE = !cycle; } - write64(ir0 + 0x18, - virt_to_phys(&raw const EVENT_RING as u64) - + (EVENT_RING_DEQUEUE as u64) * 16 - | (1 << 3)); - if ttype == trb_type::COMMAND_COMPLETION - || ttype == trb_type::NEC_CMD_COMP - { - let cc = trb.completion_code(); - if cc == completion_code::SUCCESS { - xhci_trace_note(0, "nec_fw_ok"); - } else { - xhci_trace_note(0, "nec_fw_fail"); - } - got_response = true; - break; - } - // If it's not a command completion, continue (may be port status change) - } - } - core::hint::spin_loop(); - } - if !got_response { - xhci_trace_note(0, "nec_fw_timeout"); - } + // NOTE: 3s pre-enumeration delay was tested here and did NOT fix CC=12. + // The internal reset theory has been debunked. Removed to speed boot. + + // Re-verify controller is still running after delay + let usbsts2 = read32(op_base + 0x04); + if usbsts2 & 1 != 0 { + xhci_trace_note(0, "err:halted_after_delay"); } - // 13. Create state with IRQ already set + // EXPERIMENT: Configure MSI BEFORE enumeration. + // Theory: Parallels hypervisor needs MSI configured before it will create + // internal endpoint state. On Linux, xhci_hcd always configures MSI during + // init (even if later unbound). On Breenix, UEFI never configured MSI. + let early_irq = setup_xhci_msi(pci_dev); + + // 14. Create state with IRQ already set let mut xhci_state = XhciState { base, cap_length, @@ -3817,7 +4642,7 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { max_slots: slots_en, max_ports, context_size, - irq, + irq: 0, kbd_slot: 0, kbd_endpoint: 0, kbd_nkro_endpoint: 0, @@ -3826,12 +4651,23 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { mouse_nkro_endpoint: 0, }; - // 14. Scan ports and configure HID devices. - // - // MSI is configured at PCI level (address/data written to xHC before RS=1, - // matching Linux's pci_alloc_irq_vectors). GIC SPI is NOT yet enabled — - // enumeration uses direct event ring polling via wait_for_event/wait_for_command. - if let Err(_) = scan_ports(&mut xhci_state) { + // NOOP command: warm up the command ring before real commands. + // Linux xhci_hcd sends a NEC vendor NOOP as its first command after RS=1. + // This completes a full command ring cycle (queue TRB → ring doorbell → + // receive completion event → update ERDP) before Enable Slot. + if let Err(e) = send_noop(&xhci_state) { + crate::serial_println!("[xhci] NOOP command failed: {}", e); + } + + // 15. Scan ports and configure HID devices. + if INHERIT_UEFI { + if let Err(e) = inherit_uefi_endpoints(&mut xhci_state) { + crate::serial_println!("[xhci] INHERIT_UEFI failed: {}, falling back to scan_ports", e); + if let Err(_) = scan_ports(&mut xhci_state) { + xhci_trace_note(0, "err:port_scan"); + } + } + } else if let Err(_) = scan_ports(&mut xhci_state) { xhci_trace_note(0, "err:port_scan"); } @@ -3877,28 +4713,59 @@ pub fn init(pci_dev: &crate::drivers::pci::Device) -> Result<(), &'static str> { let _mouse2_state = read_output_ep_state(s, s.mouse_slot, s.mouse_nkro_endpoint); } - // Drain stale events from enumeration BEFORE queuing interrupt TRBs. - // This prevents drain_stale_events from consuming CC=12 Transfer Events - // generated by the interrupt TRBs we're about to queue. - start_hid_polling(xhci_state_ref); - HID_POLLING_STARTED.store(true, Ordering::Release); + // NOTE: 2s delay was tested here (matching Linux's msleep(2000)) and did + // NOT fix CC=12. The issue is not timing-related. - // Clear all PORTSC change bits BEFORE queuing interrupt TRBs. - // Linux explicitly acknowledges port status changes via PORTSC W1C writes. - // The Parallels vxHC may refuse to service interrupt endpoints (CC=12) - // if port change conditions haven't been acknowledged. + // CRITICAL: Clear ALL PORTSC change bits IMMEDIATELY before queuing TRBs. + // + // Discovery: the Parallels vxHC returns CC=12 (Endpoint Not Enabled) when + // there are unacknowledged PORTSC change bits (especially CSC, bit 17). + // + // The Linux module's port reset code writes `portsc | PR` which ACCIDENTALLY + // clears all pending change bits (they're W1C — Write-1-to-Clear). Breenix's + // port reset code carefully preserves change bits with a mask, so they + // accumulate and are never cleared. + // + // Previously, PORTSC clearing was done AFTER start_hid_polling() which is + // too late — CC=12 occurs immediately when the doorbell is rung. The clearing + // must happen BEFORE any doorbell ring for interrupt endpoints. let ports_cleared = clear_all_port_changes(xhci_state_ref); DIAG_PORTSC_CLEARED.store(ports_cleared, Ordering::Relaxed); + xhci_trace_note(0, "portsc_cleared"); + + // Delay before queueing TRBs: the Linux kernel module has a 2-second + // msleep(2000) between ConfigureEndpoint completion and first doorbell ring. + // This may allow the Parallels virtual xHC to stabilize internal state. + // POST_ENUM_DELAY_MS controls this delay (0 = no delay). + if POST_ENUM_DELAY_MS > 0 { + delay_ms(POST_ENUM_DELAY_MS); + } - // NO immediate TRB queue. The Parallels vxHC fires an internal XHC reset - // ~400ms after HCRST that destroys endpoint state (despite output context - // still reading "Running"). TRBs are queued in the deferred poll=600 path - // (~3s after timer starts) which re-issues ConfigureEndpoint first to - // re-create endpoints after the internal reset settles. + // Queue TRBs immediately after PORTSC clearing + delay. + if DEFERRED_TRB_POLL == 0 { + start_hid_polling(xhci_state_ref); + HID_POLLING_STARTED.store(true, Ordering::Release); + } + // else: TRBs will be queued at poll == DEFERRED_TRB_POLL in poll_hid_events. + + // 16. MSI was already configured BEFORE enumeration (experiment). + // Store the IRQ from the early setup. + XHCI_IRQ.store(early_irq, Ordering::Release); xhci_trace_note(0, "init_complete"); - xhci_trace_dump(); - XHCI_TRACE_ACTIVE.store(false, Ordering::Relaxed); + // Trace data available via GDB (call trace_dump()) or /proc/xhci/trace + + // --- M11: EVENT_DELIVERY --- + ms_begin!(M_EVT_DELIV); + // M11 is left open — PASS/FAIL is determined by the first transfer event + // which arrives asynchronously after init() returns. The poll path will + // report M11 completion when a Transfer Event is received. + + // Keep XHCI_TRACE_ACTIVE enabled so btrace (/proc/xhci/trace) shows + // post-init events (Transfer Events, doorbell rings, etc.) + + // Single summary line for successful init + crate::serial_println!("[xhci] Initialized: {} slots, MSI irq={}", slots_en, early_irq); Ok(()) } @@ -4023,9 +4890,18 @@ pub fn handle_interrupt() { LAST_NKRO_REPORT_U64.store(nkro_snap, Ordering::Relaxed); if nkro[0] == 1 { KBD_EVENT_COUNT.fetch_add(1, Ordering::Relaxed); - super::hid::process_keyboard_report(&nkro[1..9]); + // NKRO report (after report ID) is: + // [modifier, key1, key2, ..., key7] — NO reserved byte + // Boot keyboard format expects: + // [modifier, reserved=0, key1, ..., key6] + // Reformat by inserting a zero reserved byte. + let boot_fmt: [u8; 8] = [ + nkro[1], 0, // modifier, reserved + nkro[2], nkro[3], nkro[4], nkro[5], nkro[6], nkro[7], + ]; + super::hid::process_keyboard_report(&boot_fmt); } - MSI_NKRO_NEEDS_REQUEUE.store(true, Ordering::Release); + let _ = queue_hid_transfer(state, 2, slot, endpoint); } // Boot keyboard (DCI 3, interface 0) else if slot == state.kbd_slot && endpoint == state.kbd_endpoint { @@ -4034,17 +4910,15 @@ pub fn handle_interrupt() { dma_cache_invalidate((*report_buf).0.as_ptr(), 8); let report = &(*report_buf).0; super::hid::process_keyboard_report(report); - // DON'T requeue here — let the timer poll requeue. - // Requeuing from IRQ context creates an MSI storm - // (virtual XHCI has no bus latency, so completions - // fire instantly, starving the main thread). - MSI_KBD_NEEDS_REQUEUE.store(true, Ordering::Release); + // Requeue immediately — SPI is disabled at the top + // of handle_interrupt so no MSI storm is possible. + let _ = queue_hid_transfer(state, 0, slot, endpoint); } else if slot == state.mouse_slot && endpoint == state.mouse_endpoint { let report_buf = &raw const MOUSE_REPORT_BUF; dma_cache_invalidate((*report_buf).0.as_ptr(), 8); let report = &(*report_buf).0; super::hid::process_mouse_report(report); - MSI_MOUSE_NEEDS_REQUEUE.store(true, Ordering::Release); + let _ = queue_hid_transfer(state, 1, slot, endpoint); } else if slot == state.mouse_slot && state.mouse_nkro_endpoint != 0 && endpoint == state.mouse_nkro_endpoint @@ -4054,7 +4928,7 @@ pub fn handle_interrupt() { dma_cache_invalidate((*report_buf).0.as_ptr(), 9); let report = &(*report_buf).0; super::hid::process_mouse_report(report); - MSI_MOUSE2_NEEDS_REQUEUE.store(true, Ordering::Release); + let _ = queue_hid_transfer(state, 3, slot, endpoint); } else if slot == state.mouse_slot && endpoint == 1 && HID_TRBS_QUEUED.load(Ordering::Relaxed) @@ -4144,16 +5018,8 @@ pub fn handle_interrupt() { // Polling Mode (fallback for systems without interrupt support) // ============================================================================= -/// Poll for HID events without relying on interrupts. -/// Deferred re-configuration + TRB queue. -/// -/// Called from poll_hid_events at poll=600 (~3s after timer start). -/// Re-issues ConfigureEndpoint + BW dance for both slots to re-create -/// interrupt endpoints destroyed by the Parallels vxHC internal reset, -/// then queues interrupt TRBs and rings doorbells. -/// Phase 1 of deferred init: re-issue ConfigureEndpoint + BW dance for both -/// slots, but do NOT queue TRBs yet. This is called at poll=600 (~3s after -/// timer start) to re-create endpoints after the Parallels internal XHC reset. +/// Phase 1 of deferred init (DISABLED — see comment in poll_hid_events). +#[allow(dead_code)] fn deferred_reconfigure_only(state: &XhciState) { // Drain any stale events from the event ring first. drain_stale_events(state); @@ -4172,7 +5038,7 @@ fn deferred_reconfigure_only(state: &XhciState) { b_interval: 4, ss_max_burst: 0, ss_bytes_per_interval: 64, - ss_mult: 0, + _ss_mult: 0, }); ep_count += 1; @@ -4184,7 +5050,7 @@ fn deferred_reconfigure_only(state: &XhciState) { b_interval: 4, ss_max_burst: 0, ss_bytes_per_interval: 64, - ss_mult: 0, + _ss_mult: 0, }); ep_count += 1; } @@ -4220,9 +5086,8 @@ fn deferred_reconfigure_only(state: &XhciState) { // The virtual xHC does not implement ConfigureEndpoint with DC=1, // causing wait_for_command to block forever (no Command Completion event). - match configure_endpoints_batch(state, slot_id, &pending_eps, ep_count) { - Ok(()) => crate::serial_println!("[xhci] deferred mouse slot {} cfg_ep OK (ep_count={})", slot_id, ep_count), - Err(e) => crate::serial_println!("[xhci] deferred mouse slot {} cfg_ep FAIL: {} (ep_count={})", slot_id, e, ep_count), + if let Err(e) = configure_endpoints_batch(state, slot_id, &pending_eps, ep_count) { + crate::serial_println!("[xhci] deferred mouse slot {} cfg_ep FAIL: {} (ep_count={})", slot_id, e, ep_count); } } @@ -4240,7 +5105,7 @@ fn deferred_reconfigure_only(state: &XhciState) { b_interval: 4, ss_max_burst: 0, ss_bytes_per_interval: 64, - ss_mult: 0, + _ss_mult: 0, }); ep_count += 1; @@ -4252,7 +5117,7 @@ fn deferred_reconfigure_only(state: &XhciState) { b_interval: 4, ss_max_burst: 0, ss_bytes_per_interval: 64, - ss_mult: 0, + _ss_mult: 0, }); ep_count += 1; } @@ -4285,64 +5150,21 @@ fn deferred_reconfigure_only(state: &XhciState) { // Deconfigure (DC=1) REMOVED: Parallels vxHC hangs on this command. - match configure_endpoints_batch(state, slot_id, &pending_eps, ep_count) { - Ok(()) => crate::serial_println!("[xhci] deferred kbd slot {} cfg_ep OK (ep_count={})", slot_id, ep_count), - Err(e) => crate::serial_println!("[xhci] deferred kbd slot {} cfg_ep FAIL: {} (ep_count={})", slot_id, e, ep_count), + if let Err(e) = configure_endpoints_batch(state, slot_id, &pending_eps, ep_count) { + crate::serial_println!("[xhci] deferred kbd slot {} cfg_ep FAIL: {} (ep_count={})", slot_id, e, ep_count); } } - // Dump EP states from output context to verify Running after deferred cfg_ep. - unsafe { - for &(slot_id, label) in &[(state.mouse_slot, "mouse"), (state.kbd_slot, "kbd")] { - if slot_id != 0 { - let slot_idx = (slot_id - 1) as usize; - let dev_ctx = &raw const DEVICE_CONTEXTS[slot_idx]; - dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); - let ctx_size = state.context_size; - for dci in [3u8, 5u8] { - let ep_dw0 = core::ptr::read_volatile( - (*dev_ctx).0.as_ptr().add(dci as usize * ctx_size) as *const u32, - ); - let ep_state = ep_dw0 & 0x7; - crate::serial_println!( - "[xhci] deferred {} slot {} DCI {} EP_state={} (0=Disabled,1=Running)", - label, slot_id, dci, ep_state - ); - } - } - } - } } /// Phase 2 of deferred init: queue TRBs and ring doorbells for all HID /// endpoints. Called at poll=1200 (~6s after timer start), 3 seconds after /// the deferred ConfigureEndpoint to allow any secondary internal reset to settle. +#[allow(dead_code)] fn deferred_queue_trbs(state: &XhciState) { // Drain any events that appeared between ConfigEP and now. drain_stale_events(state); - // Dump EP states right before queuing to verify they're still Running. - unsafe { - for &(slot_id, label) in &[(state.mouse_slot, "mouse"), (state.kbd_slot, "kbd")] { - if slot_id != 0 { - let slot_idx = (slot_id - 1) as usize; - let dev_ctx = &raw const DEVICE_CONTEXTS[slot_idx]; - dma_cache_invalidate((*dev_ctx).0.as_ptr(), 4096); - let ctx_size = state.context_size; - for dci in [3u8, 5u8] { - let ep_dw0 = core::ptr::read_volatile( - (*dev_ctx).0.as_ptr().add(dci as usize * ctx_size) as *const u32, - ); - let ep_state = ep_dw0 & 0x7; - crate::serial_println!( - "[xhci] pre-queue {} slot {} DCI {} EP_state={} (0=Disabled,1=Running)", - label, slot_id, dci, ep_state - ); - } - } - } - } - KBD_TRB_FIRST_QUEUED.store(true, Ordering::Release); HID_TRBS_QUEUED.store(true, Ordering::Release); @@ -4375,6 +5197,13 @@ pub fn poll_hid_events() { POLL_COUNT.fetch_add(1, Ordering::Relaxed); + // Rate-limit: poll every 4th tick (~50 Hz at 200 Hz timer). + // Balances responsiveness (20ms latency) with hypervisor overhead. + let poll = POLL_COUNT.load(Ordering::Relaxed); + if poll % 4 != 0 { + return; + } + // try_lock: if someone else holds the lock, skip this poll cycle let _guard = match XHCI_LOCK.try_lock() { Some(g) => g, @@ -4400,6 +5229,16 @@ pub fn poll_hid_events() { write32(state.op_base + 0x04, 1 << 3); // W1C to clear EINT } + // Deferred TRB queue: queue interrupt TRBs after a delay (matching Linux's msleep). + let poll = POLL_COUNT.load(Ordering::Relaxed); + if DEFERRED_TRB_POLL > 0 + && poll == DEFERRED_TRB_POLL + && !HID_POLLING_STARTED.load(Ordering::Acquire) + { + start_hid_polling(state); + HID_POLLING_STARTED.store(true, Ordering::Release); + } + // Process all pending events on the event ring loop { unsafe { @@ -4508,13 +5347,17 @@ pub fn poll_hid_events() { ]); LAST_NKRO_REPORT_U64.store(nkro_snap, Ordering::Relaxed); - // NKRO reports have Report ID prefix: - // [report_id=1, modifiers, reserved, key1..key6] = 9 bytes - // Strip the Report ID byte and pass the standard 8-byte - // boot keyboard report format to process_keyboard_report. + // NKRO report (after report ID) has NO reserved byte: + // [report_id=1, modifier, key1, key2, ..., key7] = 9 bytes + // Reformat to boot keyboard layout for process_keyboard_report: + // [modifier, reserved=0, key1, ..., key6] if nkro[0] == 1 { KBD_EVENT_COUNT.fetch_add(1, Ordering::Relaxed); - super::hid::process_keyboard_report(&nkro[1..9]); + let boot_fmt: [u8; 8] = [ + nkro[1], 0, // modifier, reserved + nkro[2], nkro[3], nkro[4], nkro[5], nkro[6], nkro[7], + ]; + super::hid::process_keyboard_report(&boot_fmt); } let _ = queue_hid_transfer(state, 2, state.kbd_slot, state.kbd_nkro_endpoint); @@ -4570,24 +5413,35 @@ pub fn poll_hid_events() { ((slot as u64) << 16) | ((endpoint as u64) << 8) | (cc as u64), Ordering::Relaxed, ); - // Read endpoint state from output context on first error CC. - // Diagnostic: tells us the state after CC=12 (Running=1, Halted=2, etc.) + // Read full endpoint + slot context on first error CC. if DIAG_EP_STATE_AFTER_CC12.load(Ordering::Relaxed) == 0xFF && slot > 0 && (slot as usize) <= MAX_SLOTS { let slot_idx = (slot - 1) as usize; - let ep_out = DEVICE_CONTEXTS[slot_idx] - .0 - .as_ptr() - .add(endpoint as usize * state.context_size); - dma_cache_invalidate(ep_out, 4); + let ctx_base = DEVICE_CONTEXTS[slot_idx].0.as_ptr(); + dma_cache_invalidate(ctx_base, 4096); + let ep_out = ctx_base.add(endpoint as usize * state.context_size); let dw0 = core::ptr::read_volatile(ep_out as *const u32); let ep_state = dw0 & 0x7; DIAG_EP_STATE_AFTER_CC12.store( ((slot as u32) << 16) | ((endpoint as u32) << 8) | ep_state, Ordering::Relaxed, ); + // Capture full EP context DW0-DW4 + DIAG_CC12_EP_DW0.store(dw0, Ordering::Relaxed); + DIAG_CC12_EP_DW1.store(core::ptr::read_volatile(ep_out.add(4) as *const u32), Ordering::Relaxed); + DIAG_CC12_EP_DW2.store(core::ptr::read_volatile(ep_out.add(8) as *const u32), Ordering::Relaxed); + DIAG_CC12_EP_DW3.store(core::ptr::read_volatile(ep_out.add(12) as *const u32), Ordering::Relaxed); + DIAG_CC12_EP_DW4.store(core::ptr::read_volatile(ep_out.add(16) as *const u32), Ordering::Relaxed); + // Capture Slot Context DW0 and DW3 + DIAG_CC12_SLOT_DW0.store(core::ptr::read_volatile(ctx_base as *const u32), Ordering::Relaxed); + DIAG_CC12_SLOT_DW3.store(core::ptr::read_volatile(ctx_base.add(12) as *const u32), Ordering::Relaxed); + // Capture DCBAA entry for this slot + let dcbaa = &raw const DCBAA; + dma_cache_invalidate( + &(*dcbaa).0[slot as usize] as *const u64 as *const u8, 8); + DIAG_CC12_DCBAA.store((*dcbaa).0[slot as usize], Ordering::Relaxed); } if slot == state.kbd_slot && endpoint == state.kbd_endpoint { NEEDS_RESET_KBD_BOOT.store(true, Ordering::Release); @@ -4633,13 +5487,28 @@ pub fn poll_hid_events() { } } - // MSI requeue flags: clear without requeuing. The reset state machine - // below already requeues after resetting, and poll_hid_events requeues - // after successful transfers. Double-requeuing would cause duplicate TRBs. - let _ = MSI_KBD_NEEDS_REQUEUE.swap(false, Ordering::AcqRel); - let _ = MSI_NKRO_NEEDS_REQUEUE.swap(false, Ordering::AcqRel); - let _ = MSI_MOUSE_NEEDS_REQUEUE.swap(false, Ordering::AcqRel); - let _ = MSI_MOUSE2_NEEDS_REQUEUE.swap(false, Ordering::AcqRel); + // MSI requeue fallback: if the MSI handler failed to requeue (e.g., lock + // contention caused the handler to bail early), requeue here as a safety net. + if MSI_KBD_NEEDS_REQUEUE.swap(false, Ordering::AcqRel) { + if state.kbd_slot != 0 && state.kbd_endpoint != 0 { + let _ = queue_hid_transfer(state, 0, state.kbd_slot, state.kbd_endpoint); + } + } + if MSI_NKRO_NEEDS_REQUEUE.swap(false, Ordering::AcqRel) { + if state.kbd_slot != 0 && state.kbd_nkro_endpoint != 0 { + let _ = queue_hid_transfer(state, 2, state.kbd_slot, state.kbd_nkro_endpoint); + } + } + if MSI_MOUSE_NEEDS_REQUEUE.swap(false, Ordering::AcqRel) { + if state.mouse_slot != 0 && state.mouse_endpoint != 0 { + let _ = queue_hid_transfer(state, 1, state.mouse_slot, state.mouse_endpoint); + } + } + if MSI_MOUSE2_NEEDS_REQUEUE.swap(false, Ordering::AcqRel) { + if state.mouse_slot != 0 && state.mouse_nkro_endpoint != 0 { + let _ = queue_hid_transfer(state, 3, state.mouse_slot, state.mouse_nkro_endpoint); + } + } let poll = POLL_COUNT.load(Ordering::Relaxed); diff --git a/kernel/src/drivers/virtio/gpu_pci.rs b/kernel/src/drivers/virtio/gpu_pci.rs index 2adddda3..4aae2b9f 100644 --- a/kernel/src/drivers/virtio/gpu_pci.rs +++ b/kernel/src/drivers/virtio/gpu_pci.rs @@ -313,19 +313,10 @@ pub fn init() -> Result<(), &'static str> { return Ok(()); } - crate::serial_println!("[virtio-gpu-pci] Searching for GPU PCI device..."); - // Find VirtIO GPU PCI device (device_id 0x1050 = 0x1040 + 16) let pci_dev = crate::drivers::pci::find_device(0x1AF4, 0x1050) .ok_or("No VirtIO GPU PCI device found")?; - crate::serial_println!( - "[virtio-gpu-pci] Found GPU at PCI {:02x}:{:02x}.{:x}", - pci_dev.bus, - pci_dev.device, - pci_dev.function - ); - // Probe VirtIO modern transport let mut virtio = VirtioPciDevice::probe(pci_dev) .ok_or("VirtIO GPU PCI: no modern capabilities")?; @@ -336,17 +327,11 @@ pub fn init() -> Result<(), &'static str> { // state-modifying commands (create_resource, attach_backing, etc.). let requested = VIRTIO_F_VERSION_1 | VIRTIO_GPU_F_EDID; virtio.init(requested)?; - let dev_feats = virtio.device_features(); - let negotiated = dev_feats & requested; - crate::serial_println!( - "[virtio-gpu-pci] Features: device={:#x} requested={:#x} negotiated={:#x}", - dev_feats, requested, negotiated - ); + let _negotiated = virtio.device_features() & requested; // Set up control queue (queue 0) virtio.select_queue(0); let queue_max = virtio.get_queue_num_max(); - crate::serial_println!("[virtio-gpu-pci] Control queue max size: {}", queue_max); if queue_max == 0 { return Err("Control queue size is 0"); @@ -395,27 +380,8 @@ pub fn init() -> Result<(), &'static str> { // If create_resource/attach_backing/set_scanout/flush time out, leaving // the flag true would mislead other code into thinking the device is usable. - // Log physical addresses for diagnostics (DMA correctness) - let fb_virt = &raw const PCI_FRAMEBUFFER as u64; - let fb_phys = virt_to_phys(fb_virt); - let cmd_virt = &raw const PCI_CMD_BUF as u64; - let cmd_phys = virt_to_phys(cmd_virt); - let queue_virt = &raw const PCI_CTRL_QUEUE as u64; - let queue_phys = virt_to_phys(queue_virt); - crate::serial_println!( - "[virtio-gpu-pci] DMA addrs: fb={:#x}->{:#x} cmd={:#x}->{:#x} queue={:#x}->{:#x}", - fb_virt, fb_phys, cmd_virt, cmd_phys, queue_virt, queue_phys - ); - - // Query display info for diagnostics. - match get_display_info() { - Ok((w, h)) => { - crate::serial_println!("[virtio-gpu-pci] Display reports: {}x{}", w, h); - } - Err(e) => { - crate::serial_println!("[virtio-gpu-pci] get_display_info failed: {}", e); - } - } + // Query display info (ignore result — we override to our desired resolution). + let _ = get_display_info(); // Override to our desired resolution. // On Parallels, VirtIO GPU set_scanout controls the display MODE (stride, @@ -423,7 +389,6 @@ pub fn init() -> Result<(), &'static str> { // 0x10000000). We use VirtIO GPU purely to configure a higher resolution // than the GOP-reported 1024x768. let (use_width, use_height) = (DEFAULT_FB_WIDTH, DEFAULT_FB_HEIGHT); - crate::serial_println!("[virtio-gpu-pci] Requesting resolution: {}x{}", use_width, use_height); // Update state with actual dimensions unsafe { @@ -611,17 +576,10 @@ fn get_display_info() -> Result<(u32, u32), &'static str> { return Err("GET_DISPLAY_INFO failed"); } - // Log ALL scanouts for diagnostics let mut first_enabled = None; - for (i, pmode) in resp.pmodes.iter().enumerate() { - if pmode.r_width > 0 || pmode.r_height > 0 || pmode.enabled != 0 { - crate::serial_println!( - "[virtio-gpu-pci] Scanout {}: {}x{} enabled={} flags={:#x}", - i, pmode.r_width, pmode.r_height, pmode.enabled, pmode.flags - ); - if pmode.enabled != 0 && first_enabled.is_none() { - first_enabled = Some((pmode.r_width, pmode.r_height)); - } + for (_i, pmode) in resp.pmodes.iter().enumerate() { + if pmode.enabled != 0 && first_enabled.is_none() { + first_enabled = Some((pmode.r_width, pmode.r_height)); } } diff --git a/kernel/src/fs/procfs/mod.rs b/kernel/src/fs/procfs/mod.rs index c0216413..cbb28975 100644 --- a/kernel/src/fs/procfs/mod.rs +++ b/kernel/src/fs/procfs/mod.rs @@ -42,6 +42,7 @@ use alloc::vec::Vec; use spin::Mutex; mod trace; +mod xhci; /// Procfs entry types #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -82,6 +83,10 @@ pub enum ProcEntryType { Pids, /// /proc/kmsg - kernel log messages Kmsg, + /// /proc/xhci - xHCI directory + XhciDir, + /// /proc/xhci/trace - xHCI trace buffer + XhciTrace, /// /proc/[pid] - per-process directory (dynamic, not registered) PidDir(u64), /// /proc/[pid]/status - per-process status (dynamic, not registered) @@ -113,6 +118,8 @@ impl ProcEntryType { ProcEntryType::BreenixTesting => "testing", ProcEntryType::Pids => "pids", ProcEntryType::Kmsg => "kmsg", + ProcEntryType::XhciDir => "xhci", + ProcEntryType::XhciTrace => "trace", ProcEntryType::PidDir(_) => "pid", ProcEntryType::PidStatus(_) => "status", } @@ -142,6 +149,8 @@ impl ProcEntryType { ProcEntryType::BreenixTesting => "/proc/breenix/testing", ProcEntryType::Pids => "/proc/pids", ProcEntryType::Kmsg => "/proc/kmsg", + ProcEntryType::XhciDir => "/proc/xhci", + ProcEntryType::XhciTrace => "/proc/xhci/trace", // Dynamic entries don't have static paths ProcEntryType::PidDir(_) => "/proc/", ProcEntryType::PidStatus(_) => "/proc//status", @@ -173,6 +182,8 @@ impl ProcEntryType { ProcEntryType::BreenixTesting => 201, ProcEntryType::Pids => 9, ProcEntryType::Kmsg => 10, + ProcEntryType::XhciDir => 300, + ProcEntryType::XhciTrace => 301, ProcEntryType::PidDir(pid) => 10000 + pid, ProcEntryType::PidStatus(pid) => 20000 + pid, } @@ -182,7 +193,10 @@ impl ProcEntryType { pub fn is_directory(&self) -> bool { matches!( self, - ProcEntryType::TraceDir | ProcEntryType::BreenixDir | ProcEntryType::PidDir(_) + ProcEntryType::TraceDir + | ProcEntryType::BreenixDir + | ProcEntryType::XhciDir + | ProcEntryType::PidDir(_) ) } } @@ -245,6 +259,10 @@ pub fn init() { procfs.entries.push(ProcEntry::new(ProcEntryType::Pids)); procfs.entries.push(ProcEntry::new(ProcEntryType::Kmsg)); + // Register /proc/xhci directory and entries + procfs.entries.push(ProcEntry::new(ProcEntryType::XhciDir)); + procfs.entries.push(ProcEntry::new(ProcEntryType::XhciTrace)); + // Register /proc/trace directory and entries procfs.entries.push(ProcEntry::new(ProcEntryType::TraceDir)); procfs.entries.push(ProcEntry::new(ProcEntryType::TraceEnable)); @@ -335,6 +353,7 @@ pub fn list_entries() -> Vec { | ProcEntryType::TraceCounters | ProcEntryType::TraceProviders | ProcEntryType::BreenixTesting + | ProcEntryType::XhciTrace )) .map(|e| String::from(e.entry_type.name())) .collect() @@ -372,6 +391,17 @@ pub fn list_trace_entries() -> Vec { .collect() } +/// List entries in the /proc/xhci directory +pub fn list_xhci_entries() -> Vec { + let procfs = PROCFS.lock(); + procfs + .entries + .iter() + .filter(|e| matches!(e.entry_type, ProcEntryType::XhciTrace)) + .map(|e| String::from(e.entry_type.name())) + .collect() +} + /// List entries in the /proc/breenix directory pub fn list_breenix_entries() -> Vec { let procfs = PROCFS.lock(); @@ -417,6 +447,11 @@ pub fn read_entry(entry_type: ProcEntryType) -> Result { ProcEntryType::Pids => Ok(generate_pids()), ProcEntryType::Kmsg => Ok(generate_kmsg()), ProcEntryType::Mounts => Ok(generate_mounts()), + ProcEntryType::XhciDir => { + let entries = list_xhci_entries(); + Ok(entries.join("\n") + "\n") + } + ProcEntryType::XhciTrace => Ok(xhci::generate_xhci_trace()), ProcEntryType::BreenixDir => { // Directory listing Ok(String::from("testing\n")) diff --git a/kernel/src/fs/procfs/xhci.rs b/kernel/src/fs/procfs/xhci.rs new file mode 100644 index 00000000..1180d345 --- /dev/null +++ b/kernel/src/fs/procfs/xhci.rs @@ -0,0 +1,11 @@ +//! Procfs generator for /proc/xhci/trace +//! +//! Exposes the xHCI binary trace buffer through procfs so userspace +//! programs (like btrace) can read it. + +use alloc::string::String; + +/// Generate the content of /proc/xhci/trace by reading the static xHCI trace buffers. +pub fn generate_xhci_trace() -> String { + crate::drivers::usb::xhci::format_trace_buffer() +} diff --git a/kernel/src/main_aarch64.rs b/kernel/src/main_aarch64.rs index e3890bf0..68712347 100644 --- a/kernel/src/main_aarch64.rs +++ b/kernel/src/main_aarch64.rs @@ -280,6 +280,22 @@ pub extern "C" fn kernel_main(hw_config_ptr: u64) -> ! { } } + // Zero the .dma section (Non-Cacheable DMA buffer region). + // This memory is NOLOAD in the ELF, so neither the loader nor boot.S zeroes it. + // Must happen before any driver init that uses DMA buffers. + unsafe { + extern "C" { + static __dma_start: u8; + static __dma_end: u8; + } + let start = &__dma_start as *const u8 as *mut u8; + let end = &__dma_end as *const u8; + let len = end as usize - start as usize; + if len > 0 && len < 0x20_0000 { + core::ptr::write_bytes(start, 0, len); + } + } + // Install the kernel's exception vector table (VBAR_EL1). // On QEMU, boot.S already did this before jumping to kernel_main. // On Parallels, the UEFI loader installed minimal "write X and spin" vectors. @@ -314,6 +330,14 @@ pub extern "C" fn kernel_main(hw_config_ptr: u64) -> ! { serial_println!("========================================"); serial_println!(); + // Diagnostic: verify this code is reached (no format args = no alloc issues) + serial_println!("[boot] DIAG_MARKER_XHCI_A"); + let hcrst_raw = kernel::platform_config::xhci_hcrst_done_raw(); + serial_println!("[boot] DIAG_MARKER_XHCI_B"); + let ecam = kernel::platform_config::pci_ecam_base(); + serial_println!("[boot] DIAG_MARKER_XHCI_C"); + serial_println!("[boot] loader xhci_hcrst_raw=0x{:x} ecam=0x{:x}", hcrst_raw, ecam); + // Print CPU info let el = current_exception_level(); serial_println!("[boot] Current exception level: EL{}", el); diff --git a/kernel/src/memory/heap.rs b/kernel/src/memory/heap.rs index 51a3b59a..963b3fca 100644 --- a/kernel/src/memory/heap.rs +++ b/kernel/src/memory/heap.rs @@ -12,19 +12,18 @@ pub const HEAP_START: u64 = 0x_4444_4444_0000; // ARM64 heap uses the direct-mapped region from boot.S (TTBR1 high-half). // The heap MUST be in TTBR1 because TTBR0 gets switched to process page tables. // -// boot.S maps TTBR1 L1[1] = physical 0x4000_0000..0x7FFF_FFFF to virtual 0xFFFF_0000_4000_0000.. -// Frame allocator uses: physical 0x4200_0000 to 0x5000_0000 -// Heap must be placed AFTER the frame allocator to avoid collision! -pub const HEAP_START: u64 = crate::arch_impl::aarch64::constants::HHDM_BASE + 0x5000_0000; +// Memory layout (physical): +// Frame allocator: 0x4200_0000 - 0x5000_0000 +// .dma (NC) block: 0x5000_0000 - 0x501F_FFFF (2 MB, Non-Cacheable for xHCI DMA) +// Heap: 0x5020_0000 - 0x51FF_FFFF (30 MB, Write-Back Cacheable) +// Kernel stacks: 0x5200_0000 - 0x53FF_FFFF (32 MB) +// +// The heap MUST start AFTER the 2 MB NC DMA block to avoid overlapping +// with xHCI DMA buffers placed in the .dma linker section. +pub const HEAP_START: u64 = crate::arch_impl::aarch64::constants::HHDM_BASE + 0x5020_0000; -/// Heap size: 32 MiB. -/// -/// This provides sufficient headroom for: -/// - Boot initialization allocations -/// - Running 10+ concurrent processes with full fd tables -/// - ext2 filesystem operations -/// - Network stack buffers -pub const HEAP_SIZE: u64 = 32 * 1024 * 1024; +/// Heap size: 30 MiB (reduced from 32 to make room for 2 MB NC DMA block). +pub const HEAP_SIZE: u64 = 30 * 1024 * 1024; /// Global allocator instance using a proper free-list allocator. /// diff --git a/kernel/src/platform_config.rs b/kernel/src/platform_config.rs index 896dcdc4..7bdecb51 100644 --- a/kernel/src/platform_config.rs +++ b/kernel/src/platform_config.rs @@ -61,11 +61,18 @@ static PCI_BUS_START: AtomicU64 = AtomicU64::new(0); #[cfg(target_arch = "aarch64")] static PCI_BUS_END: AtomicU64 = AtomicU64::new(255); +// xHCI loader-level HCRST flag. Non-zero if the parallels-loader already did +// HCRST before ExitBootServices (kernel should skip HCRST). +#[cfg(target_arch = "aarch64")] +static XHCI_HCRST_DONE: AtomicU64 = AtomicU64::new(0); + // Memory layout defaults (QEMU virt, 512MB RAM at 0x40000000) // Kernel image: 0x4000_0000 - 0x4100_0000 (16 MB) // Per-CPU stacks: 0x4100_0000 - 0x4200_0000 (16 MB) // Frame alloc: 0x4200_0000 - 0x5000_0000 (224 MB) -// Heap: 0x5000_0000 - 0x5200_0000 (32 MB) +// DMA (NC): 0x5000_0000 - 0x501F_FFFF (2 MB, Non-Cacheable for xHCI) +// Heap: 0x5020_0000 - 0x51FF_FFFF (30 MB) +// Kernel stacks: 0x5200_0000 - 0x5400_0000 (32 MB) #[cfg(target_arch = "aarch64")] static FRAME_ALLOC_START: AtomicU64 = AtomicU64::new(0x4200_0000); @@ -280,6 +287,26 @@ pub fn is_qemu() -> bool { uart_base_phys() == 0x0900_0000 } +/// Whether the parallels-loader already performed HCRST before ExitBootServices. +/// If true, the kernel should skip HCRST in xhci::init to avoid destroying +/// endpoint state that was created while the xHCI BAR was still active. +#[cfg(target_arch = "aarch64")] +#[inline] +pub fn xhci_hcrst_done() -> bool { + // The loader now disconnects UEFI's xHCI driver instead of doing HCRST. + // The kernel should always do its own HCRST. Return false unconditionally. + false +} + +/// Raw value of the xHCI HCRST sentinel (for diagnostics). +/// Sentinels: 0x00=untouched, 0xEE=reached, 0xCC=no ECAM, 0xDD=VID mismatch, +/// 0xBB=BAR zero, 0x01=HCRST done +#[cfg(target_arch = "aarch64")] +#[inline] +pub fn xhci_hcrst_done_raw() -> u64 { + XHCI_HCRST_DONE.load(Ordering::Relaxed) +} + /// Whether a UEFI GOP framebuffer was discovered by the loader. #[cfg(target_arch = "aarch64")] #[inline] @@ -365,6 +392,9 @@ pub struct HardwareConfig { pub framebuffer: FramebufferInfo, pub rsdp_addr: u64, pub timer_freq_hz: u64, + pub xhci_hcrst_done: u32, + pub _pad6: u32, + pub xhci_bar_phys: u64, } #[cfg(target_arch = "aarch64")] @@ -452,8 +482,8 @@ pub fn init_from_parallels(config: &HardwareConfig) -> bool { if best_size > 0 { // Frame allocator starts after kernel + stacks (32 MB from RAM base) let fa_start = best_base + 0x0200_0000; // +32 MB - // Frame allocator must end BEFORE the heap region. - // The heap is at fixed physical 0x5000_0000 (32 MB), so cap fa_end there. + // Frame allocator must end BEFORE the DMA NC region. + // The .dma section starts at physical 0x5000_0000, so cap fa_end there. let fa_end = (best_base + best_size).min(0x5000_0000); if fa_end > fa_start { FRAME_ALLOC_START.store(fa_start, Ordering::Relaxed); @@ -472,5 +502,10 @@ pub fn init_from_parallels(config: &HardwareConfig) -> bool { FB_IS_BGR.store(if config.framebuffer.pixel_format == 1 { 1 } else { 0 }, Ordering::Relaxed); } + // Store xHCI loader-level HCRST flag + if config.xhci_hcrst_done != 0 { + XHCI_HCRST_DONE.store(config.xhci_hcrst_done as u64, Ordering::Relaxed); + } + true } diff --git a/kernel/src/serial_aarch64.rs b/kernel/src/serial_aarch64.rs index 2d148c91..d9a00b70 100644 --- a/kernel/src/serial_aarch64.rs +++ b/kernel/src/serial_aarch64.rs @@ -367,7 +367,7 @@ pub fn raw_serial_str(s: &[u8]) { #[macro_export] macro_rules! serial_print { ($($arg:tt)*) => { - $crate::serial_aarch64::_print(format_args!($($arg)*)); + $crate::serial_aarch64::_print(format_args!($($arg)*)) }; } diff --git a/linux_xhci_module/Makefile b/linux_xhci_module/Makefile new file mode 100644 index 00000000..725fbc47 --- /dev/null +++ b/linux_xhci_module/Makefile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Out-of-tree build for breenix_xhci_probe Linux kernel module. +# Build on the target VM with: make +# Cross-compile with: make KDIR=/path/to/kernel/build + +ifneq ($(KERNELRELEASE),) +obj-m := breenix_xhci_probe.o +ccflags-y := -DDEBUG +else +KDIR ?= /lib/modules/$(shell uname -r)/build +default: + $(MAKE) -C $(KDIR) M=$(PWD) modules +clean: + $(MAKE) -C $(KDIR) M=$(PWD) clean +endif diff --git a/linux_xhci_module/breenix_xhci_probe.c b/linux_xhci_module/breenix_xhci_probe.c new file mode 100644 index 00000000..3451fe7a --- /dev/null +++ b/linux_xhci_module/breenix_xhci_probe.c @@ -0,0 +1,2546 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * breenix_xhci_probe - Standalone xHCI probe module for Breenix validation. + * + * This is a plain PCI driver that claims an xHCI controller, does raw + * register-level init, enumerates USB devices, configures HID interrupt + * endpoints, and prints received HID reports to dmesg. + * + * It does NOT use the Linux USB HCD framework (usb_hcd / hc_driver). + * Its purpose is to validate xHCI logic independently of Breenix kernel + * infrastructure (DMA mapping, cache coherency, memory model). + * + * Ported from kernel/src/drivers/usb/linux_xhci/linux_xhci.c (Breenix). + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* ----------------------------------------------------------------------- + * Constants + * ----------------------------------------------------------------------- */ +#define MAX_SLOTS 32 +#define MAX_HID_EPS 4 +#define MAX_INTR_ENDPOINTS (MAX_SLOTS * MAX_HID_EPS) +#define MAX_PORTS 256 +#define TRBS_PER_SEGMENT 256 +#define SEGMENT_POOL_COUNT 64 +#define RING_POOL_COUNT 16 +#define CTRL_BUF_SIZE 512 +#define INTR_BUF_SIZE 1024 + +/* TRB types */ +#define TRB_TYPE_NORMAL 1 +#define TRB_TYPE_SETUP 2 +#define TRB_TYPE_DATA 3 +#define TRB_TYPE_STATUS 4 +#define TRB_TYPE_LINK 6 +#define TRB_TYPE_NOOP 8 +#define TRB_TYPE_ENABLE_SLOT 9 +#define TRB_TYPE_ADDRESS_DEVICE 11 +#define TRB_TYPE_CONFIGURE_ENDPOINT 12 +#define TRB_TYPE_STOP_ENDPOINT 15 +#define TRB_TYPE_SET_TR_DEQ 16 +#define TRB_TYPE_TRANSFER_EVENT 32 +#define TRB_TYPE_COMMAND_COMPLETION 33 + +/* TRB control bits */ +#define TRB_CYCLE (1u << 0) +#define TRB_TC (1u << 1) +#define TRB_ISP (1u << 2) +#define TRB_IOC (1u << 5) +#define TRB_IDT (1u << 6) +#define TRB_DIR_IN (1u << 16) +#define TRB_TRT_SHIFT 16 +#define TRB_SLOT_ID_SHIFT 24 +#define TRB_EP_ID_SHIFT 16 + +/* Slot/EP context field macros */ +#define LAST_CTX(p) ((p) << 27) +#define ROOT_HUB_PORT(p) (((p) & 0xffu) << 16) +#define EP_MULT(p) (((p) & 0x3u) << 8) +#define EP_INTERVAL(p) (((p) & 0xffu) << 16) +#define EP_TYPE(p) ((p) << 3) +#define ERROR_COUNT(p) (((p) & 0x3u) << 1) +#define MAX_BURST(p) (((p) & 0xffu) << 8) +#define MAX_PACKET(p) (((p) & 0xffffu) << 16) +#define EP_AVG_TRB_LENGTH(p) ((p) & 0xffffu) +#define EP_MAX_ESIT_PAYLOAD_LO(p) (((p) & 0xffffu) << 16) +#define EP_MAX_ESIT_PAYLOAD_HI(p) ((((p) >> 16) & 0xffu) << 24) + +#define SLOT_SPEED_SS (4u << 20) +#define SLOT_SPEED_SSP (5u << 20) +#define SLOT_SPEED_HS (3u << 20) +#define SLOT_SPEED_FS (2u << 20) +#define SLOT_SPEED_LS (1u << 20) + +#define EP0_FLAG (1u << 1) +#define SLOT_FLAG (1u << 0) +#define CTRL_EP 4 +#define INT_IN_EP 7 + +#define CC_SUCCESS 1 +#define CC_SHORT_PACKET 13 + +/* Operational registers (offsets from op_base) */ +#define USBCMD_OFF 0x00 +#define USBSTS_OFF 0x04 +#define DNCTRL_OFF 0x14 +#define CRCR_OFF 0x18 +#define DCBAAP_OFF 0x30 +#define CONFIG_OFF 0x38 + +/* PORTSC bits */ +#define PORTSC_CCS (1u << 0) +#define PORTSC_PED (1u << 1) +#define PORTSC_PR (1u << 4) +#define PORTSC_PRC (1u << 21) +#define PORTSC_SPEED_SHIFT 10 +#define PORTSC_SPEED_MASK (0xFu << PORTSC_SPEED_SHIFT) + +/* Interrupter registers (offsets from ir0_base) */ +#define IMAN_OFF 0x00 +#define IMOD_OFF 0x04 +#define ERSTSZ_OFF 0x08 +#define ERSTBA_OFF 0x10 +#define ERDP_OFF 0x18 + +/* USB speeds */ +#define USB_SPEED_LOW 1 +#define USB_SPEED_FULL 2 +#define USB_SPEED_HIGH 3 +#define USB_SPEED_SUPER 4 +#define USB_SPEED_SUPER_PLUS 5 + +/* USB descriptor types */ +#define USB_DT_INTERFACE 4 +#define USB_DT_ENDPOINT 5 +#define USB_DT_HID 0x21 +#define USB_DT_SS_EP_COMP 0x30 +#define USB_CLASS_HID 0x03 + +/* USB endpoint transfer types */ +#define USB_EP_XFER_CONTROL 0 +#define USB_EP_XFER_ISOC 1 +#define USB_EP_XFER_BULK 2 +#define USB_EP_XFER_INT 3 + +/* ----------------------------------------------------------------------- + * Data structures + * ----------------------------------------------------------------------- */ + +struct xhci_trb { + u32 field[4]; +}; + +struct xhci_segment { + struct xhci_trb *trbs; /* virtual */ + dma_addr_t dma; /* physical */ + struct xhci_segment *next; +}; + +enum xhci_ring_type { + TYPE_COMMAND = 0, + TYPE_EVENT = 1, + TYPE_CTRL = 2, + TYPE_INTR = 3, +}; + +struct xhci_ring { + struct xhci_segment *first_seg; + struct xhci_segment *last_seg; + struct xhci_segment *enq_seg; + struct xhci_trb *enqueue; + u32 cycle_state; + unsigned int num_segs; + enum xhci_ring_type type; +}; + +struct xhci_slot_ctx { + u32 dev_info; + u32 dev_info2; + u32 tt_info; + u32 dev_state; + u32 reserved[4]; +}; + +struct xhci_ep_ctx { + u32 ep_info; + u32 ep_info2; + u64 deq; + u32 tx_info; + u32 reserved[3]; +}; + +struct xhci_input_control_ctx { + u32 drop_flags; + u32 add_flags; + u32 rsvd2[6]; +}; + +struct xhci_erst_entry { + u64 seg_addr; + u32 seg_size; + u32 rsvd; +}; + +struct xhci_virt_device { + u8 slot_id; + u8 ctx_size; + u8 *in_ctx; + dma_addr_t in_ctx_dma; + u8 *reconfig_in_ctx; + dma_addr_t reconfig_in_ctx_dma; + u8 *out_ctx; + dma_addr_t out_ctx_dma; + struct xhci_ring *ep_rings[32]; +}; + +/* USB descriptor structs (packed) */ +struct usb_config_desc { + u8 bLength; + u8 bDescriptorType; + __le16 wTotalLength; + u8 bNumInterfaces; + u8 bConfigurationValue; + u8 iConfiguration; + u8 bmAttributes; + u8 bMaxPower; +} __packed; + +struct usb_iface_desc { + u8 bLength; + u8 bDescriptorType; + u8 bInterfaceNumber; + u8 bAlternateSetting; + u8 bNumEndpoints; + u8 bInterfaceClass; + u8 bInterfaceSubClass; + u8 bInterfaceProtocol; + u8 iInterface; +} __packed; + +struct usb_ep_desc { + u8 bLength; + u8 bDescriptorType; + u8 bEndpointAddress; + u8 bmAttributes; + __le16 wMaxPacketSize; + u8 bInterval; +} __packed; + +struct usb_ss_ep_comp_desc { + u8 bLength; + u8 bDescriptorType; + u8 bMaxBurst; + u8 bmAttributes; + __le16 wBytesPerInterval; +} __packed; + +struct usb_hid_desc { + u8 bLength; + u8 bDescriptorType; + __le16 bcdHID; + u8 bCountryCode; + u8 bNumDescriptors; + u8 bReportDescriptorType; + __le16 wDescriptorLength; +} __packed; + +struct usb_setup_packet { + u8 bmRequestType; + u8 bRequest; + __le16 wValue; + __le16 wIndex; + __le16 wLength; +} __packed; + +struct host_endpoint { + struct usb_ep_desc desc; + struct usb_ss_ep_comp_desc ss_ep_comp; + u8 iface_num; + u8 iface_subclass; + u8 iface_protocol; + u16 report_len; +}; + +struct usb_dev_min { + u8 speed; + u8 slot_id; + u8 portnum; + u32 route; +}; + +struct intr_ep_queue { + u8 slot_id; + u8 dci; + struct xhci_ring *ep_ring; + u32 max_packet; +}; + +/* ----------------------------------------------------------------------- + * Per-device probe state (replaces all globals from C harness) + * ----------------------------------------------------------------------- */ +struct probe_state { + struct pci_dev *pdev; + void __iomem *bar; /* ioremapped BAR0 */ + size_t bar_len; + + /* Capability/operational/runtime/doorbell base offsets from bar */ + u32 op_off; + u32 rt_off; + u32 db_off; + u8 max_slots; + u8 max_ports; + u8 ctx_size; + u16 hci_version; + + /* DCBAA */ + u64 *dcbaa; + dma_addr_t dcbaa_dma; + + /* ERST */ + struct xhci_erst_entry *erst; + dma_addr_t erst_dma; + + /* Segment pool */ + struct xhci_segment segments[SEGMENT_POOL_COUNT]; + struct xhci_trb *seg_trb_va[SEGMENT_POOL_COUNT]; + dma_addr_t seg_trb_dma[SEGMENT_POOL_COUNT]; + unsigned int seg_alloc_idx; + + /* Rings */ + struct xhci_ring cmd_ring; + struct xhci_ring event_ring; + + /* Event ring dequeue state */ + struct xhci_segment *event_deq_seg; + struct xhci_trb *event_dequeue; + u32 event_cycle; + + /* Virtual devices */ + struct xhci_virt_device virt_devs[MAX_SLOTS]; + + /* EP0 ring pool */ + struct xhci_ring ep0_ring_pool[MAX_SLOTS]; + + /* Interrupt endpoint ring pool */ + struct xhci_ring ring_pool[RING_POOL_COUNT]; + unsigned int ring_pool_idx; + + /* Control transfer buffer */ + u8 *ctrl_buf; + dma_addr_t ctrl_buf_dma; + + /* Interrupt transfer buffers */ + u8 *intr_bufs[MAX_INTR_ENDPOINTS]; + dma_addr_t intr_bufs_dma[MAX_INTR_ENDPOINTS]; + + /* Interrupt endpoint tracking */ + struct intr_ep_queue intr_eps[MAX_INTR_ENDPOINTS]; + unsigned int intr_count; + + /* MSI IRQ */ + int irq; +}; + +/* ----------------------------------------------------------------------- + * MMIO write trace buffer (HCRST through first doorbell ring) + * ----------------------------------------------------------------------- */ +#define MMIO_TRACE_MAX 512 + +struct mmio_trace_entry { + u32 offset; /* offset from BAR base */ + u32 value; /* 32-bit value written */ + u32 seq; /* monotonically increasing sequence number */ +}; + +static struct mmio_trace_entry mmio_trace_buf[MMIO_TRACE_MAX]; +static u32 mmio_trace_idx; /* next write index */ +static u32 mmio_trace_seq; /* sequence counter */ +static bool mmio_trace_active; + +static inline void mmio_trace_record(u32 offset, u32 value) +{ + if (!mmio_trace_active) + return; + if (mmio_trace_idx < MMIO_TRACE_MAX) { + mmio_trace_buf[mmio_trace_idx].offset = offset; + mmio_trace_buf[mmio_trace_idx].value = value; + mmio_trace_buf[mmio_trace_idx].seq = mmio_trace_seq++; + mmio_trace_idx++; + } +} + +/* ----------------------------------------------------------------------- + * MMIO helpers + * ----------------------------------------------------------------------- */ +static inline u32 xhci_read32(struct probe_state *st, u32 offset) +{ + return readl(st->bar + offset); +} + +static inline void xhci_write32(struct probe_state *st, u32 offset, u32 val) +{ + mmio_trace_record(offset, val); + writel(val, st->bar + offset); +} + +static inline u64 xhci_read64(struct probe_state *st, u32 offset) +{ + u32 lo = readl(st->bar + offset); + u32 hi = readl(st->bar + offset + 4); + return ((u64)hi << 32) | lo; +} + +static inline void xhci_write64(struct probe_state *st, u32 offset, u64 val) +{ + mmio_trace_record(offset, (u32)(val & 0xFFFFFFFF)); + mmio_trace_record(offset + 4, (u32)(val >> 32)); + writel((u32)(val & 0xFFFFFFFF), st->bar + offset); + writel((u32)(val >> 32), st->bar + offset + 4); +} + +/* Convenience: operational register access */ +static inline u32 op_read32(struct probe_state *st, u32 reg) +{ + return xhci_read32(st, st->op_off + reg); +} + +static inline void op_write32(struct probe_state *st, u32 reg, u32 val) +{ + xhci_write32(st, st->op_off + reg, val); +} + +static inline u64 op_read64(struct probe_state *st, u32 reg) +{ + return xhci_read64(st, st->op_off + reg); +} + +static inline void op_write64(struct probe_state *st, u32 reg, u64 val) +{ + xhci_write64(st, st->op_off + reg, val); +} + +/* Convenience: interrupter 0 register access */ +static inline u32 ir0_off(struct probe_state *st) +{ + return st->rt_off + 0x20; +} + +static inline u32 ir0_read32(struct probe_state *st, u32 reg) +{ + return xhci_read32(st, ir0_off(st) + reg); +} + +static inline void ir0_write32(struct probe_state *st, u32 reg, u32 val) +{ + xhci_write32(st, ir0_off(st) + reg, val); +} + +static inline u64 ir0_read64(struct probe_state *st, u32 reg) +{ + return xhci_read64(st, ir0_off(st) + reg); +} + +static inline void ir0_write64(struct probe_state *st, u32 reg, u64 val) +{ + xhci_write64(st, ir0_off(st) + reg, val); +} + +/* Doorbell */ +static inline void ring_doorbell(struct probe_state *st, u8 slot, u8 target) +{ + xhci_write32(st, st->db_off + (u32)slot * 4, target); +} + +/* ----------------------------------------------------------------------- + * Milestone-based initialization outline + * + * The xHCI initialization is organized into numbered milestones. Each + * milestone validates specific invariants. When comparing Linux vs Breenix, + * the first milestone that diverges identifies the root cause. + * + * M1: CONTROLLER_DISCOVERY — BAR mapped, capabilities read + * M2: CONTROLLER_RESET — HCRST done, CNR clear, HCH=1 + * M3: DATA_STRUCTURES — DCBAA, CMD ring, EVT ring, ERST programmed + * M4: CONTROLLER_RUNNING — RS=1, INTE=1, IMAN.IE=1 + * M5: PORT_DETECTION — Connected ports identified, speed known + * M6: SLOT_ENABLE — EnableSlot CC=1, slot ID allocated + * M7: DEVICE_ADDRESS — Input ctx built, AddressDevice CC=1 + * M8: ENDPOINT_CONFIG — ConfigureEndpoint CC=1, BW dance done + * M9: HID_CLASS_SETUP — SET_CONFIGURATION, SET_IDLE, descriptors + * M10: INTERRUPT_TRANSFER — Normal TRBs queued, doorbells rung + * M11: EVENT_DELIVERY — First transfer event (HID data received) + * ----------------------------------------------------------------------- */ + +#define M_DISCOVERY 1 +#define M_RESET 2 +#define M_DATA_STRUC 3 +#define M_RUNNING 4 +#define M_PORT_DET 5 +#define M_SLOT_EN 6 +#define M_ADDR_DEV 7 +#define M_EP_CONFIG 8 +#define M_HID_SETUP 9 +#define M_INTR_XFER 10 +#define M_EVT_DELIV 11 +#define M_TOTAL 11 + +static const char * const milestone_names[] = { + [0] = "UNUSED", + [1] = "CONTROLLER_DISCOVERY", + [2] = "CONTROLLER_RESET", + [3] = "DATA_STRUCTURES", + [4] = "CONTROLLER_RUNNING", + [5] = "PORT_DETECTION", + [6] = "SLOT_ENABLE", + [7] = "DEVICE_ADDRESS", + [8] = "ENDPOINT_CONFIG", + [9] = "HID_CLASS_SETUP", + [10] = "INTERRUPT_TRANSFER", + [11] = "EVENT_DELIVERY", +}; + +static void ms_begin(struct probe_state *st, int m) +{ + dev_info(&st->pdev->dev, + "=== MILESTONE %d/%d: %s ===\n", m, M_TOTAL, + milestone_names[m]); +} + +static void ms_pass(struct probe_state *st, int m) +{ + dev_info(&st->pdev->dev, + "[M%d] RESULT: PASS\n", m); +} + +static void ms_fail(struct probe_state *st, int m, const char *reason) +{ + dev_info(&st->pdev->dev, + "[M%d] RESULT: FAIL (%s)\n", m, reason); +} + +/* Print a key=value pair under a milestone */ +#define ms_kv(st, m, fmt, ...) \ + dev_info(&(st)->pdev->dev, "[M%d] " fmt "\n", (m), ##__VA_ARGS__) + +/* Hex-dump a DMA buffer under a milestone */ +static void ms_dump(struct probe_state *st, int m, const char *label, + const void *buf, dma_addr_t dma, size_t len) +{ + const u32 *p = buf; + size_t i, n; + + dev_info(&st->pdev->dev, "[M%d] %s: dma=0x%llx len=%zu\n", + m, label, (u64)dma, len); + n = len / 4; + for (i = 0; i < n; i += 4) { + if (i + 3 < n) { + dev_info(&st->pdev->dev, + "[M%d] +%03zx: %08x %08x %08x %08x\n", + m, i * 4, p[i], p[i+1], p[i+2], p[i+3]); + } else if (n - i == 3) { + dev_info(&st->pdev->dev, + "[M%d] +%03zx: %08x %08x %08x\n", + m, i * 4, p[i], p[i+1], p[i+2]); + } else if (n - i == 2) { + dev_info(&st->pdev->dev, + "[M%d] +%03zx: %08x %08x\n", + m, i * 4, p[i], p[i+1]); + } else if (n - i == 1) { + dev_info(&st->pdev->dev, + "[M%d] +%03zx: %08x\n", + m, i * 4, p[i]); + } + } +} + +/* Dump a TRB under a milestone */ +static void ms_trb(struct probe_state *st, int m, const char *label, + const struct xhci_trb *trb) +{ + dev_info(&st->pdev->dev, + "[M%d] %s: %08x %08x %08x %08x\n", + m, label, trb->field[0], trb->field[1], + trb->field[2], trb->field[3]); +} + +/* Dump all key registers under a milestone */ +static void ms_regs(struct probe_state *st, int m) +{ + ms_kv(st, m, "USBCMD=0x%08x USBSTS=0x%08x", + op_read32(st, USBCMD_OFF), op_read32(st, USBSTS_OFF)); + ms_kv(st, m, "DCBAAP=0x%016llx CRCR=0x%016llx", + (u64)op_read64(st, DCBAAP_OFF), (u64)op_read64(st, CRCR_OFF)); + ms_kv(st, m, "IMAN=0x%08x IMOD=0x%08x ERSTSZ=0x%08x", + ir0_read32(st, IMAN_OFF), ir0_read32(st, IMOD_OFF), + ir0_read32(st, ERSTSZ_OFF)); + ms_kv(st, m, "ERDP=0x%016llx ERSTBA=0x%016llx", + (u64)ir0_read64(st, ERDP_OFF), (u64)ir0_read64(st, ERSTBA_OFF)); +} + +/* Dump full 256-byte PCI config space in milestone format for comparison */ +static void ms_pci_config(struct probe_state *st, int m, const char *label) +{ + int offset; + + for (offset = 0; offset < 256; offset += 16) { + u32 dw0, dw1, dw2, dw3; + + pci_read_config_dword(st->pdev, offset, &dw0); + pci_read_config_dword(st->pdev, offset + 4, &dw1); + pci_read_config_dword(st->pdev, offset + 8, &dw2); + pci_read_config_dword(st->pdev, offset + 12, &dw3); + dev_info(&st->pdev->dev, + "[M%d] %s +%03x: %08x %08x %08x %08x\n", + m, label, offset, dw0, dw1, dw2, dw3); + } +} + +/* ----------------------------------------------------------------------- + * USB endpoint helpers + * ----------------------------------------------------------------------- */ +static inline u8 ep_type(const struct usb_ep_desc *d) +{ + return d->bmAttributes & 0x3; +} + +static inline bool ep_is_int(const struct usb_ep_desc *d) +{ + return ep_type(d) == USB_EP_XFER_INT; +} + +static inline bool ep_is_bulk(const struct usb_ep_desc *d) +{ + return ep_type(d) == USB_EP_XFER_BULK; +} + +static inline bool ep_is_isoc(const struct usb_ep_desc *d) +{ + return ep_type(d) == USB_EP_XFER_ISOC; +} + +static inline bool ep_is_ctrl(const struct usb_ep_desc *d) +{ + return ep_type(d) == USB_EP_XFER_CONTROL; +} + +static inline bool ep_dir_in(const struct usb_ep_desc *d) +{ + return (d->bEndpointAddress & 0x80) != 0; +} + +static inline u8 ep_num(const struct usb_ep_desc *d) +{ + return d->bEndpointAddress & 0x0F; +} + +static inline unsigned int ep_maxp(const struct usb_ep_desc *d) +{ + return le16_to_cpu(d->wMaxPacketSize) & 0x7FF; +} + +static inline unsigned int ep_maxp_mult(const struct usb_ep_desc *d) +{ + return ((le16_to_cpu(d->wMaxPacketSize) >> 11) & 0x3) + 1; +} + +/* ----------------------------------------------------------------------- + * Context helpers + * ----------------------------------------------------------------------- */ +static inline struct xhci_input_control_ctx *get_input_control_ctx(u8 *ctx) +{ + return (struct xhci_input_control_ctx *)ctx; +} + +static inline struct xhci_slot_ctx *get_slot_ctx_in(u8 *ctx, size_t ctx_size) +{ + return (struct xhci_slot_ctx *)(ctx + ctx_size); +} + +static inline struct xhci_slot_ctx *get_slot_ctx_out(u8 *ctx) +{ + return (struct xhci_slot_ctx *)ctx; +} + +static inline struct xhci_ep_ctx *get_ep_ctx_in(u8 *ctx, size_t ctx_size, u8 dci) +{ + return (struct xhci_ep_ctx *)(ctx + (1 + (size_t)dci) * ctx_size); +} + +static inline struct xhci_ep_ctx *get_ep_ctx_out(u8 *ctx, size_t ctx_size, u8 dci) +{ + return (struct xhci_ep_ctx *)(ctx + (size_t)dci * ctx_size); +} + +/* ----------------------------------------------------------------------- + * Interval calculation (matches Linux xhci-mem.c) + * ----------------------------------------------------------------------- */ +static unsigned int fls_u32(u32 v) +{ + unsigned int r = 0; + + while (v) { + v >>= 1; + r++; + } + return r; +} + +static unsigned int xhci_parse_exponent_interval(const struct host_endpoint *ep) +{ + unsigned int bi = ep->desc.bInterval; + + if (bi < 1) + bi = 1; + if (bi > 16) + bi = 16; + return bi - 1; +} + +static unsigned int xhci_microframes_to_exponent(unsigned int desc_interval, + unsigned int min_exp, + unsigned int max_exp) +{ + unsigned int interval = fls_u32(desc_interval) - 1; + + return clamp(interval, min_exp, max_exp); +} + +static unsigned int xhci_parse_microframe_interval(const struct host_endpoint *ep) +{ + if (ep->desc.bInterval == 0) + return 0; + return xhci_microframes_to_exponent(ep->desc.bInterval, 0, 15); +} + +static unsigned int xhci_parse_frame_interval(const struct host_endpoint *ep) +{ + return xhci_microframes_to_exponent((unsigned int)ep->desc.bInterval * 8, 3, 10); +} + +static unsigned int xhci_get_endpoint_interval(const struct usb_dev_min *udev, + const struct host_endpoint *ep) +{ + unsigned int interval = 0; + + switch (udev->speed) { + case USB_SPEED_HIGH: + if (ep_is_ctrl(&ep->desc) || ep_is_bulk(&ep->desc)) { + interval = xhci_parse_microframe_interval(ep); + break; + } + fallthrough; + case USB_SPEED_SUPER_PLUS: + case USB_SPEED_SUPER: + if (ep_is_int(&ep->desc) || ep_is_isoc(&ep->desc)) + interval = xhci_parse_exponent_interval(ep); + break; + case USB_SPEED_FULL: + if (ep_is_isoc(&ep->desc)) { + interval = xhci_parse_exponent_interval(ep); + break; + } + fallthrough; + case USB_SPEED_LOW: + if (ep_is_int(&ep->desc) || ep_is_isoc(&ep->desc)) + interval = xhci_parse_frame_interval(ep); + break; + default: + break; + } + return interval; +} + +static unsigned int usb_ep_max_periodic_payload(const struct usb_dev_min *udev, + const struct host_endpoint *ep) +{ + if (ep_is_ctrl(&ep->desc) || ep_is_bulk(&ep->desc)) + return 0; + if (udev->speed >= USB_SPEED_SUPER) { + unsigned int bytes = le16_to_cpu(ep->ss_ep_comp.wBytesPerInterval); + + if (bytes == 0) { + unsigned int mp = ep_maxp(&ep->desc); + unsigned int mb = ep->ss_ep_comp.bMaxBurst; + unsigned int mult = (ep->ss_ep_comp.bmAttributes & 0x3) + 1; + + bytes = mp * (mb + 1) * mult; + } + return bytes; + } + return ep_maxp(&ep->desc) * ep_maxp_mult(&ep->desc); +} + +static unsigned int xhci_get_endpoint_max_burst(const struct usb_dev_min *udev, + const struct host_endpoint *ep) +{ + if (udev->speed >= USB_SPEED_SUPER) + return ep->ss_ep_comp.bMaxBurst; + if (udev->speed == USB_SPEED_HIGH && ep_is_int(&ep->desc)) + return ep_maxp_mult(&ep->desc) - 1; + return 0; +} + +static unsigned int xhci_get_endpoint_type(const struct host_endpoint *ep) +{ + int in = ep_dir_in(&ep->desc); + + if (ep_type(&ep->desc) == USB_EP_XFER_INT) + return in ? INT_IN_EP : 3; + return 0; +} + +/* ----------------------------------------------------------------------- + * Ring allocation and operations + * ----------------------------------------------------------------------- */ +static struct xhci_segment *segment_alloc(struct probe_state *st) +{ + struct xhci_segment *seg; + struct xhci_trb *trbs; + dma_addr_t dma; + + if (st->seg_alloc_idx >= SEGMENT_POOL_COUNT) { + dev_err(&st->pdev->dev, "segment pool exhausted\n"); + return NULL; + } + + trbs = dma_alloc_coherent(&st->pdev->dev, + TRBS_PER_SEGMENT * sizeof(struct xhci_trb), + &dma, GFP_KERNEL); + if (!trbs) + return NULL; + + seg = &st->segments[st->seg_alloc_idx]; + st->seg_trb_va[st->seg_alloc_idx] = trbs; + st->seg_trb_dma[st->seg_alloc_idx] = dma; + st->seg_alloc_idx++; + + dev_info(&st->pdev->dev, + "DMA alloc: virt=%px dma=0x%llx size=%zu (segment)\n", + trbs, (u64)dma, + TRBS_PER_SEGMENT * sizeof(struct xhci_trb)); + + memset(trbs, 0, TRBS_PER_SEGMENT * sizeof(struct xhci_trb)); + seg->trbs = trbs; + seg->dma = dma; + seg->next = NULL; + return seg; +} + +static void xhci_link_segment(struct xhci_segment *seg, + struct xhci_segment *next, + bool toggle_cycle) +{ + struct xhci_trb *link = &seg->trbs[TRBS_PER_SEGMENT - 1]; + + memset(link, 0, sizeof(*link)); + link->field[0] = lower_32_bits(next->dma); + link->field[1] = upper_32_bits(next->dma); + link->field[3] = (TRB_TYPE_LINK << 10) | TRB_CYCLE | + (toggle_cycle ? TRB_TC : 0); +} + +static int xhci_ring_init(struct probe_state *st, struct xhci_ring *ring, + unsigned int num_segs, enum xhci_ring_type type) +{ + struct xhci_segment *first = NULL, *prev = NULL, *cur; + unsigned int i; + + ring->num_segs = num_segs; + ring->type = type; + ring->cycle_state = 1; + ring->first_seg = NULL; + ring->last_seg = NULL; + ring->enq_seg = NULL; + ring->enqueue = NULL; + + if (num_segs == 0) + return 0; + + for (i = 0; i < num_segs; i++) { + struct xhci_segment *seg = segment_alloc(st); + + if (!seg) + return -ENOMEM; + if (!first) + first = seg; + if (prev) + prev->next = seg; + prev = seg; + } + prev->next = first; + + ring->first_seg = first; + ring->last_seg = prev; + ring->enq_seg = first; + ring->enqueue = first->trbs; + + cur = first; + for (i = 0; i < num_segs; i++) { + bool toggle = (cur == ring->last_seg); + + xhci_link_segment(cur, cur->next, toggle); + cur = cur->next; + } + return 0; +} + +static void xhci_ring_enqueue_trb(struct xhci_ring *ring, const struct xhci_trb *src) +{ + struct xhci_trb *trb = ring->enqueue; + + memcpy(trb, src, sizeof(*trb)); + if (ring->cycle_state) + trb->field[3] |= TRB_CYCLE; + else + trb->field[3] &= ~TRB_CYCLE; + + /* Advance enqueue pointer */ + if (trb == &ring->enq_seg->trbs[TRBS_PER_SEGMENT - 2]) { + /* Next is link TRB slot, move to next segment */ + ring->enq_seg = ring->enq_seg->next; + ring->enqueue = ring->enq_seg->trbs; + ring->cycle_state ^= 1; + } else { + ring->enqueue = trb + 1; + } +} + +/* ----------------------------------------------------------------------- + * Event ring handling + * ----------------------------------------------------------------------- */ +static void advance_event_dequeue(struct probe_state *st) +{ + if (st->event_dequeue == &st->event_deq_seg->trbs[TRBS_PER_SEGMENT - 1]) { + st->event_deq_seg = st->event_deq_seg->next; + st->event_dequeue = st->event_deq_seg->trbs; + st->event_cycle ^= 1; + } else { + st->event_dequeue++; + } +} + +static void ack_event(struct probe_state *st) +{ + dma_addr_t erdp; + + /* Calculate DMA address of new dequeue pointer */ + /* event_dequeue points into event_deq_seg->trbs. Offset: */ + ptrdiff_t off = st->event_dequeue - st->event_deq_seg->trbs; + + erdp = st->event_deq_seg->dma + off * sizeof(struct xhci_trb); + /* Write ERDP with EHB (bit 3) set to clear Event Handler Busy */ + ir0_write64(st, ERDP_OFF, erdp | (1ULL << 3)); + /* Clear IMAN.IP (W1C bit 0, preserve IE bit 1) */ + ir0_write32(st, IMAN_OFF, ir0_read32(st, IMAN_OFF) | 0x1); + /* Clear USBSTS.EINT (W1C bit 3) */ + op_write32(st, USBSTS_OFF, op_read32(st, USBSTS_OFF) | (1u << 3)); +} + +static int xhci_wait_for_event(struct probe_state *st, struct xhci_trb *out, + u32 expected_type) +{ + unsigned int timeout = 2000000; + + while (timeout--) { + struct xhci_trb trb = *st->event_dequeue; + u32 cycle = trb.field[3] & TRB_CYCLE; + + if ((cycle ? 1u : 0u) == st->event_cycle) { + u32 trb_type; + + advance_event_dequeue(st); + ack_event(st); + + trb_type = (trb.field[3] >> 10) & 0x3f; + if (expected_type == 0 || trb_type == expected_type) { + *out = trb; + return 0; + } + /* Unexpected type — log and continue */ + dev_dbg(&st->pdev->dev, + "skip event type=%u (expected %u)\n", + trb_type, expected_type); + } + cpu_relax(); + } + return -ETIMEDOUT; +} + +static int xhci_wait_for_event_ms(struct probe_state *st, struct xhci_trb *out, + u32 expected_type, unsigned long timeout_ms) +{ + unsigned long deadline = jiffies + msecs_to_jiffies(timeout_ms); + + while (time_before(jiffies, deadline)) { + struct xhci_trb trb = *st->event_dequeue; + u32 cycle = trb.field[3] & TRB_CYCLE; + + if ((cycle ? 1u : 0u) == st->event_cycle) { + u32 trb_type; + + advance_event_dequeue(st); + ack_event(st); + + trb_type = (trb.field[3] >> 10) & 0x3f; + if (expected_type == 0 || trb_type == expected_type) { + *out = trb; + return 0; + } + } + usleep_range(100, 500); + } + return -ETIMEDOUT; +} + +/* ----------------------------------------------------------------------- + * Command submission + * ----------------------------------------------------------------------- */ +static int xhci_cmd(struct probe_state *st, struct xhci_trb *cmd_trb, + struct xhci_trb *ev_out) +{ + xhci_ring_enqueue_trb(&st->cmd_ring, cmd_trb); + ring_doorbell(st, 0, 0); + return xhci_wait_for_event(st, ev_out, TRB_TYPE_COMMAND_COMPLETION); +} + +/* ----------------------------------------------------------------------- + * Control transfers (EP0) + * ----------------------------------------------------------------------- */ +static int control_transfer(struct probe_state *st, u8 slot_id, + struct xhci_ring *ring, + const struct usb_setup_packet *setup, + dma_addr_t data_dma, u16 data_len, bool dir_in) +{ + struct xhci_trb trb, ev; + u64 setup_data = 0; + u32 trt; + + /* Setup Stage (IDT) */ + memset(&trb, 0, sizeof(trb)); + memcpy(&setup_data, setup, sizeof(*setup)); + trb.field[0] = lower_32_bits(setup_data); + trb.field[1] = upper_32_bits(setup_data); + trb.field[2] = 8; + if (data_len == 0) + trt = 0; + else if (dir_in) + trt = 3; + else + trt = 2; + trb.field[3] = (TRB_TYPE_SETUP << 10) | TRB_IDT | (trt << TRB_TRT_SHIFT); + xhci_ring_enqueue_trb(ring, &trb); + + /* Data Stage (if any) */ + if (data_len > 0) { + memset(&trb, 0, sizeof(trb)); + trb.field[0] = lower_32_bits(data_dma); + trb.field[1] = upper_32_bits(data_dma); + trb.field[2] = data_len; + trb.field[3] = (TRB_TYPE_DATA << 10) | (dir_in ? TRB_DIR_IN : 0); + xhci_ring_enqueue_trb(ring, &trb); + } + + /* Status Stage */ + memset(&trb, 0, sizeof(trb)); + trb.field[3] = (TRB_TYPE_STATUS << 10) | TRB_IOC | + (dir_in ? 0 : TRB_DIR_IN); + xhci_ring_enqueue_trb(ring, &trb); + + ring_doorbell(st, slot_id, 1); + + if (xhci_wait_for_event(st, &ev, TRB_TYPE_TRANSFER_EVENT) != 0) + return -ETIMEDOUT; + + { + u32 cc = (ev.field[2] >> 24) & 0xFF; + + if (cc != CC_SUCCESS && cc != CC_SHORT_PACKET) { + dev_warn(&st->pdev->dev, + "ctrl xfer slot=%u CC=%u\n", slot_id, cc); + } + } + return 0; +} + +/* ----------------------------------------------------------------------- + * Endpoint setup (Linux-style context programming) + * ----------------------------------------------------------------------- */ +static int xhci_endpoint_init(struct probe_state *st, + struct xhci_virt_device *virt_dev, + const struct usb_dev_min *udev, + const struct host_endpoint *ep) +{ + u8 epn = ep_num(&ep->desc); + u8 dci = (u8)(epn * 2 + (ep_dir_in(&ep->desc) ? 1 : 0)); + struct xhci_ep_ctx *ep_ctx; + unsigned int endpoint_type, max_esit, interval, max_packet, max_burst, avg_trb_len; + struct xhci_ring *ring; + + ep_ctx = get_ep_ctx_in(virt_dev->in_ctx, virt_dev->ctx_size, dci); + + endpoint_type = xhci_get_endpoint_type(ep); + if (!endpoint_type) + return -EINVAL; + + max_esit = usb_ep_max_periodic_payload(udev, ep); + interval = xhci_get_endpoint_interval(udev, ep); + max_packet = ep_maxp(&ep->desc); + max_burst = xhci_get_endpoint_max_burst(udev, ep); + avg_trb_len = max_esit; + + /* Allocate ring from pool */ + if (st->ring_pool_idx >= RING_POOL_COUNT) { + dev_err(&st->pdev->dev, "ring pool exhausted\n"); + return -ENOMEM; + } + ring = &st->ring_pool[st->ring_pool_idx++]; + if (xhci_ring_init(st, ring, 2, TYPE_INTR) != 0) + return -ENOMEM; + virt_dev->ep_rings[dci] = ring; + + ep_ctx->ep_info = EP_MAX_ESIT_PAYLOAD_HI(max_esit) | + EP_INTERVAL(interval) | + EP_MULT(0); + ep_ctx->ep_info2 = EP_TYPE(endpoint_type) | + MAX_PACKET(max_packet) | + MAX_BURST(max_burst) | + ERROR_COUNT(3); + ep_ctx->deq = ring->first_seg->dma | ring->cycle_state; + ep_ctx->tx_info = EP_MAX_ESIT_PAYLOAD_LO(max_esit) | + EP_AVG_TRB_LENGTH(avg_trb_len); + + dev_info(&st->pdev->dev, + "ep_init slot=%u dci=%u interval=%u maxp=%u burst=%u esit=%u\n", + udev->slot_id, dci, interval, max_packet, max_burst, max_esit); + dev_info(&st->pdev->dev, + "Transfer ring slot=%u dci=%u dma=0x%llx\n", + udev->slot_id, dci, (u64)ring->first_seg->dma); + return 0; +} + +/* ----------------------------------------------------------------------- + * Ring setup: DCBAA, command ring, event ring, ERST + * ----------------------------------------------------------------------- */ +static int xhci_setup_rings(struct probe_state *st) +{ + int ret; + u64 crcr; + + /* DCBAA */ + st->dcbaa = dma_alloc_coherent(&st->pdev->dev, 256 * sizeof(u64), + &st->dcbaa_dma, GFP_KERNEL); + if (!st->dcbaa) + return -ENOMEM; + memset(st->dcbaa, 0, 256 * sizeof(u64)); + op_write64(st, DCBAAP_OFF, st->dcbaa_dma); + ms_kv(st, M_DATA_STRUC, "DCBAA: dma=0x%llx written=0x%llx readback=0x%llx", + (u64)st->dcbaa_dma, (u64)st->dcbaa_dma, + (u64)op_read64(st, DCBAAP_OFF)); + + /* Command ring */ + ret = xhci_ring_init(st, &st->cmd_ring, 1, TYPE_COMMAND); + if (ret) + return ret; + crcr = st->cmd_ring.first_seg->dma | 1; + op_write64(st, CRCR_OFF, crcr); + ms_kv(st, M_DATA_STRUC, "CMD_RING: dma=0x%llx CRCR_written=0x%llx readback=0x%llx", + (u64)st->cmd_ring.first_seg->dma, (u64)crcr, + (u64)op_read64(st, CRCR_OFF)); + + /* Event ring */ + ret = xhci_ring_init(st, &st->event_ring, 1, TYPE_EVENT); + if (ret) + return ret; + st->event_deq_seg = st->event_ring.first_seg; + st->event_dequeue = st->event_ring.first_seg->trbs; + st->event_cycle = 1; + + /* ERST */ + st->erst = dma_alloc_coherent(&st->pdev->dev, + sizeof(struct xhci_erst_entry), + &st->erst_dma, GFP_KERNEL); + if (!st->erst) + return -ENOMEM; + st->erst[0].seg_addr = st->event_ring.first_seg->dma; + st->erst[0].seg_size = TRBS_PER_SEGMENT; + st->erst[0].rsvd = 0; + + ms_kv(st, M_DATA_STRUC, "EVT_RING: seg_dma=0x%llx", + (u64)st->event_ring.first_seg->dma); + ms_kv(st, M_DATA_STRUC, "ERST: dma=0x%llx seg_addr=0x%llx seg_size=%u", + (u64)st->erst_dma, (u64)st->erst[0].seg_addr, + st->erst[0].seg_size); + + ir0_write32(st, IMOD_OFF, 0x000000a0); + ir0_write32(st, ERSTSZ_OFF, 1); + /* xHCI spec 4.9.4: ERDP before ERSTBA */ + ir0_write64(st, ERDP_OFF, st->event_ring.first_seg->dma); + ir0_write64(st, ERSTBA_OFF, st->erst_dma); + ms_kv(st, M_DATA_STRUC, "ERDP: written=0x%llx readback=0x%llx", + (u64)st->event_ring.first_seg->dma, + (u64)ir0_read64(st, ERDP_OFF)); + ms_kv(st, M_DATA_STRUC, "ERSTBA: written=0x%llx readback=0x%llx", + (u64)st->erst_dma, (u64)ir0_read64(st, ERSTBA_OFF)); + + return 0; +} + +/* ----------------------------------------------------------------------- + * Slot enable / address device / configure endpoints + * ----------------------------------------------------------------------- */ +static u8 xhci_enable_slot(struct probe_state *st) +{ + struct xhci_trb trb, ev; + + memset(&trb, 0, sizeof(trb)); + trb.field[3] = (TRB_TYPE_ENABLE_SLOT << 10); + if (xhci_cmd(st, &trb, &ev) != 0) + return 0; + return (u8)((ev.field[3] >> TRB_SLOT_ID_SHIFT) & 0xff); +} + +static int xhci_address_device(struct probe_state *st, + struct xhci_virt_device *vdev, + const struct usb_dev_min *udev) +{ + struct xhci_input_control_ctx *ctrl; + struct xhci_slot_ctx *slot_ctx; + struct xhci_ep_ctx *ep0; + u32 speed_bits; + struct xhci_trb trb, ev; + + memset(vdev->in_ctx, 0, 4096); + + ctrl = get_input_control_ctx(vdev->in_ctx); + ctrl->add_flags = SLOT_FLAG | EP0_FLAG; + ctrl->drop_flags = 0; + + slot_ctx = get_slot_ctx_in(vdev->in_ctx, vdev->ctx_size); + switch (udev->speed) { + case USB_SPEED_SUPER_PLUS: speed_bits = SLOT_SPEED_SSP; break; + case USB_SPEED_SUPER: speed_bits = SLOT_SPEED_SS; break; + case USB_SPEED_HIGH: speed_bits = SLOT_SPEED_HS; break; + case USB_SPEED_FULL: speed_bits = SLOT_SPEED_FS; break; + case USB_SPEED_LOW: speed_bits = SLOT_SPEED_LS; break; + default: speed_bits = SLOT_SPEED_SS; break; + } + slot_ctx->dev_info = speed_bits | LAST_CTX(1) | (udev->route & 0xfffff); + slot_ctx->dev_info2 = ROOT_HUB_PORT(udev->portnum); + + ep0 = get_ep_ctx_in(vdev->in_ctx, vdev->ctx_size, 1); + ep0->ep_info = 0; + ep0->ep_info2 = EP_TYPE(CTRL_EP) | MAX_PACKET(512) | + MAX_BURST(0) | ERROR_COUNT(3); + ep0->deq = vdev->ep_rings[1]->first_seg->dma | + vdev->ep_rings[1]->cycle_state; + ep0->tx_info = EP_AVG_TRB_LENGTH(8); + + ms_kv(st, M_ADDR_DEV, "slot=%u port=%u speed=%u", + udev->slot_id, udev->portnum, udev->speed); + ms_dump(st, M_ADDR_DEV, "input_ctx", vdev->in_ctx, vdev->in_ctx_dma, 192); + + memset(&trb, 0, sizeof(trb)); + trb.field[0] = lower_32_bits(vdev->in_ctx_dma); + trb.field[1] = upper_32_bits(vdev->in_ctx_dma); + trb.field[3] = (TRB_TYPE_ADDRESS_DEVICE << 10) | + ((u32)udev->slot_id << TRB_SLOT_ID_SHIFT); + + ms_trb(st, M_ADDR_DEV, "cmd_trb", &trb); + + if (xhci_cmd(st, &trb, &ev) != 0) { + ms_fail(st, M_ADDR_DEV, "command failed"); + return -EIO; + } + + { + u32 cc = (ev.field[2] >> 24) & 0xFF; + + ms_trb(st, M_ADDR_DEV, "evt_trb", &ev); + ms_kv(st, M_ADDR_DEV, "CC=%u slot=%u", cc, udev->slot_id); + } + + ms_dump(st, M_ADDR_DEV, "output_ctx", vdev->out_ctx, vdev->out_ctx_dma, 192); + + return 0; +} + +static int xhci_configure_endpoints(struct probe_state *st, + struct xhci_virt_device *vdev, + const struct usb_dev_min *udev, + const struct host_endpoint *eps, + unsigned int ep_count) +{ + struct xhci_input_control_ctx *ctrl; + struct xhci_slot_ctx *slot_ctx, *out_slot; + u32 add_flags; + u8 max_dci = 1; + unsigned int i; + struct xhci_trb trb, ev; + + memset(vdev->in_ctx, 0, 4096); + + ctrl = get_input_control_ctx(vdev->in_ctx); + ctrl->drop_flags = 0; + ctrl->add_flags = SLOT_FLAG; + + for (i = 0; i < ep_count; i++) { + u8 dci = (u8)(ep_num(&eps[i].desc) * 2 + + (ep_dir_in(&eps[i].desc) ? 1 : 0)); + ctrl->add_flags |= (1u << dci); + if (dci > max_dci) + max_dci = dci; + } + add_flags = ctrl->add_flags; + + /* Copy slot context from output, zero dev_state */ + slot_ctx = get_slot_ctx_in(vdev->in_ctx, vdev->ctx_size); + out_slot = get_slot_ctx_out(vdev->out_ctx); + memcpy(slot_ctx, out_slot, sizeof(*slot_ctx)); + slot_ctx->dev_state = 0; + slot_ctx->dev_info &= ~(0x1fu << 27); + slot_ctx->dev_info |= LAST_CTX(max_dci); + + for (i = 0; i < ep_count; i++) { + int ret = xhci_endpoint_init(st, vdev, udev, &eps[i]); + + if (ret) + return ret; + } + + /* --- M8: ENDPOINT_CONFIG --- */ + ms_begin(st, M_EP_CONFIG); + ms_kv(st, M_EP_CONFIG, "slot=%u ep_count=%u add_flags=0x%x max_dci=%u", + udev->slot_id, ep_count, ctrl->add_flags, max_dci); + ms_dump(st, M_EP_CONFIG, "cfg_ep_in_ctx", vdev->in_ctx, vdev->in_ctx_dma, 256); + + memset(&trb, 0, sizeof(trb)); + trb.field[0] = lower_32_bits(vdev->in_ctx_dma); + trb.field[1] = upper_32_bits(vdev->in_ctx_dma); + trb.field[3] = (TRB_TYPE_CONFIGURE_ENDPOINT << 10) | + ((u32)udev->slot_id << TRB_SLOT_ID_SHIFT); + + ms_trb(st, M_EP_CONFIG, "cfg_ep_cmd", &trb); + + if (xhci_cmd(st, &trb, &ev) != 0) { + ms_fail(st, M_EP_CONFIG, "ConfigureEndpoint command timeout"); + return -EIO; + } + + { + u32 cc = (ev.field[2] >> 24) & 0xFF; + + ms_kv(st, M_EP_CONFIG, "ConfigureEndpoint slot=%u CC=%u add_flags=0x%x", + udev->slot_id, cc, add_flags); + } + + ms_trb(st, M_EP_CONFIG, "cfg_ep_evt", &ev); + ms_dump(st, M_EP_CONFIG, "cfg_ep_out_ctx", vdev->out_ctx, vdev->out_ctx_dma, 256); + + /* Linux-style bandwidth dance: Stop Endpoint + re-ConfigureEndpoint */ + ms_kv(st, M_EP_CONFIG, "BW_dance: slot=%u ep_count=%u", udev->slot_id, ep_count); + for (i = 0; i < ep_count; i++) { + u8 *rc_ctx = vdev->reconfig_in_ctx; + u8 dci = (u8)(ep_num(&eps[i].desc) * 2 + + (ep_dir_in(&eps[i].desc) ? 1 : 0)); + struct xhci_trb stop_trb, stop_ev; + struct xhci_input_control_ctx *rctrl; + struct xhci_slot_ctx *rc_slot; + unsigned int j; + struct xhci_trb rc_trb, rc_ev; + + /* Stop Endpoint */ + ms_kv(st, M_EP_CONFIG, "BW_stop: slot=%u dci=%u", udev->slot_id, dci); + memset(&stop_trb, 0, sizeof(stop_trb)); + stop_trb.field[3] = (TRB_TYPE_STOP_ENDPOINT << 10) | + ((u32)udev->slot_id << TRB_SLOT_ID_SHIFT) | + ((u32)dci << TRB_EP_ID_SHIFT); + ms_trb(st, M_EP_CONFIG, "bw_stop_cmd", &stop_trb); + if (xhci_cmd(st, &stop_trb, &stop_ev) != 0) { + ms_fail(st, M_EP_CONFIG, "StopEndpoint timeout"); + ms_kv(st, M_EP_CONFIG, "StopEndpoint failed slot=%u dci=%u", + udev->slot_id, dci); + return -EIO; + } + ms_trb(st, M_EP_CONFIG, "bw_stop_evt", &stop_ev); + { + u32 stop_cc = (stop_ev.field[2] >> 24) & 0xFF; + ms_kv(st, M_EP_CONFIG, "StopEndpoint slot=%u dci=%u CC=%u", + udev->slot_id, dci, stop_cc); + } + + /* Rebuild input context from output context, re-configure */ + memset(rc_ctx, 0, 4096); + rctrl = get_input_control_ctx(rc_ctx); + rctrl->drop_flags = 0; + rctrl->add_flags = add_flags; + + rc_slot = get_slot_ctx_in(rc_ctx, vdev->ctx_size); + out_slot = get_slot_ctx_out(vdev->out_ctx); + memcpy(rc_slot, out_slot, sizeof(*rc_slot)); + rc_slot->dev_state = 0; + rc_slot->dev_info &= ~(0x1fu << 27); + rc_slot->dev_info |= LAST_CTX(max_dci); + + for (j = 0; j < ep_count; j++) { + u8 ep_dci = (u8)(ep_num(&eps[j].desc) * 2 + + (ep_dir_in(&eps[j].desc) ? 1 : 0)); + struct xhci_ep_ctx *rc_ep = + get_ep_ctx_in(rc_ctx, vdev->ctx_size, ep_dci); + struct xhci_ep_ctx *out_ep = + get_ep_ctx_out(vdev->out_ctx, vdev->ctx_size, ep_dci); + memcpy(rc_ep, out_ep, sizeof(*rc_ep)); + rc_ep->ep_info &= ~0x7u; /* clear EP state bits */ + } + + ms_dump(st, M_EP_CONFIG, "bw_reconfig_in", rc_ctx, + vdev->reconfig_in_ctx_dma, 256); + + memset(&rc_trb, 0, sizeof(rc_trb)); + rc_trb.field[0] = lower_32_bits(vdev->reconfig_in_ctx_dma); + rc_trb.field[1] = upper_32_bits(vdev->reconfig_in_ctx_dma); + rc_trb.field[3] = (TRB_TYPE_CONFIGURE_ENDPOINT << 10) | + ((u32)udev->slot_id << TRB_SLOT_ID_SHIFT); + + ms_trb(st, M_EP_CONFIG, "bw_reconfig_cmd", &rc_trb); + + if (xhci_cmd(st, &rc_trb, &rc_ev) != 0) { + ms_fail(st, M_EP_CONFIG, "re-ConfigureEndpoint timeout"); + ms_kv(st, M_EP_CONFIG, "re-ConfigureEndpoint failed slot=%u dci=%u", + udev->slot_id, dci); + return -EIO; + } + + ms_trb(st, M_EP_CONFIG, "bw_reconfig_evt", &rc_ev); + ms_dump(st, M_EP_CONFIG, "bw_reconfig_out", vdev->out_ctx, + vdev->out_ctx_dma, 256); + + { + u32 cc2 = (rc_ev.field[2] >> 24) & 0xFF; + + ms_kv(st, M_EP_CONFIG, "BW_dance slot=%u dci=%u CC=%u", + udev->slot_id, dci, cc2); + } + } + ms_pass(st, M_EP_CONFIG); + + return 0; +} + +/* ----------------------------------------------------------------------- + * Config descriptor parsing + * ----------------------------------------------------------------------- */ +static unsigned int parse_hid_endpoints(const u8 *buf, unsigned int len, + struct host_endpoint *out_eps, + unsigned int max_eps) +{ + unsigned int offset = 0, count = 0; + u8 cur_iface = 0, cur_subclass = 0, cur_protocol = 0; + u16 cur_report_len = 0; + bool in_hid = false; + + while (offset + 2 <= len) { + u8 dlen = buf[offset]; + u8 dtype = buf[offset + 1]; + + if (dlen == 0) + break; + if (offset + dlen > len) + break; + + if (dtype == USB_DT_INTERFACE && + dlen >= sizeof(struct usb_iface_desc)) { + const struct usb_iface_desc *ifd = + (const struct usb_iface_desc *)(buf + offset); + in_hid = (ifd->bInterfaceClass == USB_CLASS_HID); + cur_iface = ifd->bInterfaceNumber; + cur_subclass = ifd->bInterfaceSubClass; + cur_protocol = ifd->bInterfaceProtocol; + cur_report_len = 0; + } else if (in_hid && dtype == USB_DT_HID && + dlen >= sizeof(struct usb_hid_desc)) { + const struct usb_hid_desc *hd = + (const struct usb_hid_desc *)(buf + offset); + cur_report_len = le16_to_cpu(hd->wDescriptorLength); + } else if (in_hid && dtype == USB_DT_ENDPOINT && + dlen >= sizeof(struct usb_ep_desc)) { + const struct usb_ep_desc *epd = + (const struct usb_ep_desc *)(buf + offset); + if (ep_is_int(epd) && ep_dir_in(epd) && count < max_eps) { + memcpy(&out_eps[count].desc, epd, sizeof(*epd)); + memset(&out_eps[count].ss_ep_comp, 0, + sizeof(out_eps[count].ss_ep_comp)); + out_eps[count].iface_num = cur_iface; + out_eps[count].iface_subclass = cur_subclass; + out_eps[count].iface_protocol = cur_protocol; + out_eps[count].report_len = cur_report_len; + + /* Check for SS companion */ + if (offset + dlen + 2 <= len) { + u8 ss_len = buf[offset + dlen]; + u8 ss_type = buf[offset + dlen + 1]; + + if (ss_type == USB_DT_SS_EP_COMP && + ss_len >= sizeof(struct usb_ss_ep_comp_desc)) { + memcpy(&out_eps[count].ss_ep_comp, + buf + offset + dlen, + sizeof(struct usb_ss_ep_comp_desc)); + } + } + count++; + } + } + offset += dlen; + } + return count; +} + +struct hid_iface_info { + u8 iface; + u8 subclass; + u8 protocol; + u16 report_len; +}; + +static unsigned int build_hid_interfaces(const struct host_endpoint *eps, + unsigned int ep_count, + struct hid_iface_info *out, + unsigned int max_ifaces) +{ + unsigned int count = 0, i, j; + + for (i = 0; i < ep_count; i++) { + bool seen = false; + + for (j = 0; j < count; j++) { + if (out[j].iface == eps[i].iface_num) { + seen = true; + break; + } + } + if (seen) + continue; + if (count < max_ifaces) { + out[count].iface = eps[i].iface_num; + out[count].subclass = eps[i].iface_subclass; + out[count].protocol = eps[i].iface_protocol; + out[count].report_len = eps[i].report_len; + count++; + } + } + return count; +} + +/* ----------------------------------------------------------------------- + * Port enumeration + * ----------------------------------------------------------------------- */ +static bool enumerate_port(struct probe_state *st, u8 port, + bool *port_enumerated) +{ + u32 portsc_off = st->op_off + 0x400 + (u32)port * 0x10; + u32 portsc; + u8 slot_id; + struct xhci_virt_device *vdev; + struct usb_dev_min udev; + u32 speed_val; + struct usb_setup_packet setup; + struct usb_config_desc *cfg; + u16 total_len; + struct host_endpoint eps[MAX_HID_EPS]; + unsigned int ep_count; + u8 config_value; + + portsc = xhci_read32(st, portsc_off); + if (!(portsc & PORTSC_CCS)) + return false; + + /* --- M5: PORT_DETECTION --- */ + ms_begin(st, M_PORT_DET); + ms_kv(st, M_PORT_DET, "port=%u PORTSC=0x%08x CCS=%u PED=%u", + port + 1, portsc, !!(portsc & PORTSC_CCS), + !!(portsc & PORTSC_PED)); + ms_kv(st, M_PORT_DET, "port=%u speed_raw=%u PRC=%u", + port + 1, + (portsc & PORTSC_SPEED_MASK) >> PORTSC_SPEED_SHIFT, + !!(portsc & PORTSC_PRC)); + + if (port_enumerated[port]) { + ms_kv(st, M_PORT_DET, "port=%u already_enumerated=true", port + 1); + return false; + } + + /* Reset port if not enabled */ + if (!(portsc & PORTSC_PED)) { + unsigned int i; + + ms_kv(st, M_PORT_DET, "port=%u resetting (PED=0)", port + 1); + xhci_write32(st, portsc_off, portsc | PORTSC_PR); + for (i = 0; i < 100000; i++) { + portsc = xhci_read32(st, portsc_off); + if (!(portsc & PORTSC_PR) && (portsc & PORTSC_PED)) + break; + udelay(10); + } + ms_kv(st, M_PORT_DET, "port=%u post_reset PORTSC=0x%08x PED=%u", + port + 1, portsc, !!(portsc & PORTSC_PED)); + } + portsc = xhci_read32(st, portsc_off); + { + u8 speed_raw = (portsc & PORTSC_SPEED_MASK) >> PORTSC_SPEED_SHIFT; + + ms_kv(st, M_PORT_DET, "port=%u final PORTSC=0x%08x speed=%u", + port + 1, portsc, speed_raw); + } + ms_pass(st, M_PORT_DET); + + /* --- M6: SLOT_ENABLE --- */ + ms_begin(st, M_SLOT_EN); + slot_id = xhci_enable_slot(st); + if (slot_id == 0 || slot_id > MAX_SLOTS) { + ms_fail(st, M_SLOT_EN, "EnableSlot returned 0 or out of range"); + ms_kv(st, M_SLOT_EN, "port=%u slot_id=%u", port + 1, slot_id); + return false; + } + ms_kv(st, M_SLOT_EN, "port=%u slot_id=%u", port + 1, slot_id); + ms_pass(st, M_SLOT_EN); + + vdev = &st->virt_devs[slot_id - 1]; + vdev->slot_id = slot_id; + vdev->ctx_size = st->ctx_size; + + /* Allocate DMA contexts */ + vdev->in_ctx = dma_alloc_coherent(&st->pdev->dev, 4096, + &vdev->in_ctx_dma, GFP_KERNEL); + vdev->reconfig_in_ctx = dma_alloc_coherent(&st->pdev->dev, 4096, + &vdev->reconfig_in_ctx_dma, + GFP_KERNEL); + vdev->out_ctx = dma_alloc_coherent(&st->pdev->dev, 4096, + &vdev->out_ctx_dma, GFP_KERNEL); + if (!vdev->in_ctx || !vdev->reconfig_in_ctx || !vdev->out_ctx) { + dev_err(&st->pdev->dev, "DMA alloc failed for slot %u\n", + slot_id); + return false; + } + dev_info(&st->pdev->dev, + "DMA alloc: virt=%px dma=0x%llx size=%zu (in_ctx slot=%u)\n", + vdev->in_ctx, (u64)vdev->in_ctx_dma, (size_t)4096, slot_id); + dev_info(&st->pdev->dev, + "DMA alloc: virt=%px dma=0x%llx size=%zu (reconfig_in_ctx slot=%u)\n", + vdev->reconfig_in_ctx, (u64)vdev->reconfig_in_ctx_dma, + (size_t)4096, slot_id); + dev_info(&st->pdev->dev, + "DMA alloc: virt=%px dma=0x%llx size=%zu (out_ctx slot=%u)\n", + vdev->out_ctx, (u64)vdev->out_ctx_dma, (size_t)4096, slot_id); + memset(vdev->in_ctx, 0, 4096); + memset(vdev->reconfig_in_ctx, 0, 4096); + memset(vdev->out_ctx, 0, 4096); + memset(vdev->ep_rings, 0, sizeof(vdev->ep_rings)); + + /* EP0 ring */ + if (xhci_ring_init(st, &st->ep0_ring_pool[slot_id - 1], 2, TYPE_CTRL)) + return false; + vdev->ep_rings[1] = &st->ep0_ring_pool[slot_id - 1]; + + /* Point DCBAA to output context */ + st->dcbaa[slot_id] = vdev->out_ctx_dma; + + /* Build USB device info */ + udev.slot_id = slot_id; + udev.portnum = port + 1; + udev.route = 0; + speed_val = (portsc & PORTSC_SPEED_MASK) >> PORTSC_SPEED_SHIFT; + switch (speed_val) { + case 5: udev.speed = USB_SPEED_SUPER_PLUS; break; + case 4: udev.speed = USB_SPEED_SUPER; break; + case 3: udev.speed = USB_SPEED_HIGH; break; + case 2: udev.speed = USB_SPEED_FULL; break; + case 1: udev.speed = USB_SPEED_LOW; break; + default: udev.speed = USB_SPEED_SUPER; break; + } + ms_kv(st, M_ADDR_DEV, "port=%u speed=%u slot=%u", + port + 1, udev.speed, slot_id); + + /* --- M7: DEVICE_ADDRESS --- */ + ms_begin(st, M_ADDR_DEV); + if (xhci_address_device(st, vdev, &udev) != 0) + return false; + ms_pass(st, M_ADDR_DEV); + + /* GET CONFIG descriptor header (9 bytes) */ + memset(&setup, 0, sizeof(setup)); + setup.bmRequestType = 0x80; + setup.bRequest = 0x06; + setup.wValue = cpu_to_le16(0x0200); + setup.wIndex = 0; + setup.wLength = cpu_to_le16(9); + + memset(st->ctrl_buf, 0, CTRL_BUF_SIZE); + if (control_transfer(st, slot_id, vdev->ep_rings[1], &setup, + st->ctrl_buf_dma, 9, true) != 0) { + dev_warn(&st->pdev->dev, "GET_CONFIG header failed slot=%u\n", + slot_id); + return false; + } + + cfg = (struct usb_config_desc *)st->ctrl_buf; + total_len = le16_to_cpu(cfg->wTotalLength); + if (total_len > CTRL_BUF_SIZE) + total_len = CTRL_BUF_SIZE; + + /* GET full config descriptor */ + setup.wLength = cpu_to_le16(total_len); + memset(st->ctrl_buf, 0, CTRL_BUF_SIZE); + if (control_transfer(st, slot_id, vdev->ep_rings[1], &setup, + st->ctrl_buf_dma, total_len, true) != 0) { + dev_warn(&st->pdev->dev, "GET_CONFIG full failed slot=%u\n", + slot_id); + return false; + } + + ep_count = parse_hid_endpoints(st->ctrl_buf, total_len, eps, MAX_HID_EPS); + config_value = cfg->bConfigurationValue; + + dev_info(&st->pdev->dev, + "Slot %u: config=%u total_len=%u hid_eps=%u\n", + slot_id, config_value, total_len, ep_count); + + if (ep_count > 0) { + struct hid_iface_info hid_info[MAX_HID_EPS]; + unsigned int hid_count, idx; + + if (xhci_configure_endpoints(st, vdev, &udev, eps, ep_count)) + return false; + + /* --- M9: HID_CLASS_SETUP --- */ + ms_begin(st, M_HID_SETUP); + + /* SET_CONFIGURATION */ + { + struct usb_setup_packet set_cfg = {}; + + set_cfg.bmRequestType = 0x00; + set_cfg.bRequest = 0x09; + set_cfg.wValue = cpu_to_le16(config_value); + control_transfer(st, slot_id, vdev->ep_rings[1], + &set_cfg, 0, 0, false); + ms_kv(st, M_HID_SETUP, "SET_CONFIGURATION slot=%u config=%u", + slot_id, config_value); + ms_dump(st, M_HID_SETUP, "post_set_config_out", vdev->out_ctx, + vdev->out_ctx_dma, 256); + } + + /* HID class setup per interface */ + hid_count = build_hid_interfaces(eps, ep_count, hid_info, + MAX_HID_EPS); + ms_kv(st, M_HID_SETUP, "hid_interfaces=%u slot=%u", hid_count, slot_id); + for (idx = 0; idx < hid_count; idx++) { + struct usb_setup_packet pkt = {}; + + ms_kv(st, M_HID_SETUP, "iface[%u]: num=%u subclass=%u protocol=%u report_len=%u", + idx, hid_info[idx].iface, hid_info[idx].subclass, + hid_info[idx].protocol, hid_info[idx].report_len); + + /* SET_INTERFACE (alt 0) */ + pkt.bmRequestType = 0x01; + pkt.bRequest = 0x0B; + pkt.wValue = 0; + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + control_transfer(st, slot_id, vdev->ep_rings[1], + &pkt, 0, 0, false); + ms_kv(st, M_HID_SETUP, "SET_INTERFACE slot=%u iface=%u", + slot_id, hid_info[idx].iface); + + /* SET_PROTOCOL (boot) for boot-class devices */ + if (hid_info[idx].subclass == 1) { + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0x21; + pkt.bRequest = 0x0B; + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + control_transfer(st, slot_id, vdev->ep_rings[1], + &pkt, 0, 0, false); + ms_kv(st, M_HID_SETUP, "SET_PROTOCOL (boot) slot=%u iface=%u", + slot_id, hid_info[idx].iface); + } + + /* SET_IDLE */ + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0x21; + pkt.bRequest = 0x0A; + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + control_transfer(st, slot_id, vdev->ep_rings[1], + &pkt, 0, 0, false); + ms_kv(st, M_HID_SETUP, "SET_IDLE slot=%u iface=%u", + slot_id, hid_info[idx].iface); + + /* GET HID Report Descriptor */ + if (hid_info[idx].report_len > 0) { + u16 rlen = hid_info[idx].report_len; + + if (rlen > CTRL_BUF_SIZE) + rlen = CTRL_BUF_SIZE; + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0x81; + pkt.bRequest = 0x06; + pkt.wValue = cpu_to_le16(0x2200); + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + pkt.wLength = cpu_to_le16(rlen); + memset(st->ctrl_buf, 0, CTRL_BUF_SIZE); + control_transfer(st, slot_id, vdev->ep_rings[1], + &pkt, st->ctrl_buf_dma, + rlen, true); + } + + /* Feature reports for mouse-class HID */ + if (hid_info[idx].protocol == 2) { + u8 fid = 0; + + if (hid_info[idx].iface == 0) + fid = 0x11; + else if (hid_info[idx].iface == 1) + fid = 0x12; + if (fid) { + /* GET_REPORT (feature) */ + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0xA1; + pkt.bRequest = 0x01; + pkt.wValue = cpu_to_le16((0x03 << 8) | fid); + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + pkt.wLength = cpu_to_le16(64); + memset(st->ctrl_buf, 0, CTRL_BUF_SIZE); + control_transfer(st, slot_id, + vdev->ep_rings[1], + &pkt, st->ctrl_buf_dma, + 64, true); + + /* SET_REPORT (feature) */ + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0x21; + pkt.bRequest = 0x09; + pkt.wValue = cpu_to_le16((0x03 << 8) | fid); + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + pkt.wLength = cpu_to_le16(2); + st->ctrl_buf[0] = fid; + st->ctrl_buf[1] = fid; + control_transfer(st, slot_id, + vdev->ep_rings[1], + &pkt, st->ctrl_buf_dma, + 2, false); + } + } + + /* LED/output report for keyboards */ + if (hid_info[idx].protocol == 1) { + memset(&pkt, 0, sizeof(pkt)); + pkt.bmRequestType = 0x21; + pkt.bRequest = 0x09; + pkt.wValue = cpu_to_le16(0x0200); + pkt.wIndex = cpu_to_le16(hid_info[idx].iface); + pkt.wLength = cpu_to_le16(1); + st->ctrl_buf[0] = 0; + control_transfer(st, slot_id, vdev->ep_rings[1], + &pkt, st->ctrl_buf_dma, + 1, false); + } + } + + ms_dump(st, M_HID_SETUP, "post_hid_setup_out", vdev->out_ctx, + vdev->out_ctx_dma, 256); + ms_pass(st, M_HID_SETUP); + + /* Register interrupt endpoints */ + for (idx = 0; idx < ep_count; idx++) { + u8 dci; + struct xhci_ring *ep_ring; + + if (st->intr_count >= MAX_INTR_ENDPOINTS) + break; + dci = (u8)(ep_num(&eps[idx].desc) * 2 + 1); + ep_ring = vdev->ep_rings[dci]; + if (!ep_ring) + continue; + + st->intr_eps[st->intr_count].slot_id = slot_id; + st->intr_eps[st->intr_count].dci = dci; + st->intr_eps[st->intr_count].ep_ring = ep_ring; + st->intr_eps[st->intr_count].max_packet = + ep_maxp(&eps[idx].desc); + ms_kv(st, M_INTR_XFER, "registered intr_ep[%u]: slot=%u dci=%u maxp=%u ring_dma=0x%llx", + st->intr_count, slot_id, dci, + ep_maxp(&eps[idx].desc), + (u64)ep_ring->first_seg->dma); + st->intr_count++; + } + } + + port_enumerated[port] = true; + return true; +} + +/* ----------------------------------------------------------------------- + * MSI interrupt handler + * ----------------------------------------------------------------------- */ +static void requeue_intr_trb(struct probe_state *st, unsigned int idx) +{ + struct intr_ep_queue *info = &st->intr_eps[idx]; + struct xhci_trb trb; + u32 xfer_len; + + if (!info->ep_ring || !st->intr_bufs[idx]) + return; + + xfer_len = info->max_packet; + if (xfer_len == 0) + xfer_len = 64; + if (xfer_len > INTR_BUF_SIZE) + xfer_len = INTR_BUF_SIZE; + + memset(st->intr_bufs[idx], 0xDE, INTR_BUF_SIZE); + memset(&trb, 0, sizeof(trb)); + trb.field[0] = lower_32_bits(st->intr_bufs_dma[idx]); + trb.field[1] = upper_32_bits(st->intr_bufs_dma[idx]); + trb.field[2] = xfer_len; + trb.field[3] = (TRB_TYPE_NORMAL << 10) | TRB_IOC | TRB_ISP; + xhci_ring_enqueue_trb(info->ep_ring, &trb); + ring_doorbell(st, info->slot_id, info->dci); + dev_info(&st->pdev->dev, + "Intr TRB: slot=%u dci=%u buf_dma=0x%llx len=%u\n", + info->slot_id, info->dci, + (u64)st->intr_bufs_dma[idx], xfer_len); +} + +static irqreturn_t breenix_xhci_irq(int irq, void *data) +{ + struct probe_state *st = data; + u32 usbsts; + int handled = 0; + + /* Check USBSTS.EINT */ + usbsts = op_read32(st, USBSTS_OFF); + if (!(usbsts & (1u << 3))) + return IRQ_NONE; + + /* Process all pending events */ + while (1) { + struct xhci_trb trb = *st->event_dequeue; + u32 cycle = trb.field[3] & TRB_CYCLE; + u32 trb_type, cc, slot, ep_id, xfer_len; + + if ((cycle ? 1u : 0u) != st->event_cycle) + break; + + advance_event_dequeue(st); + handled++; + + trb_type = (trb.field[3] >> 10) & 0x3f; + cc = (trb.field[2] >> 24) & 0xFF; + slot = (trb.field[3] >> TRB_SLOT_ID_SHIFT) & 0xFF; + ep_id = (trb.field[3] >> TRB_EP_ID_SHIFT) & 0x1F; + + if (trb_type == TRB_TYPE_TRANSFER_EVENT) { + /* M11: EVENT_DELIVERY — transfer event received */ + dev_info(&st->pdev->dev, + "[M%d] xfer_event: %08x %08x %08x %08x\n", + M_EVT_DELIV, trb.field[0], trb.field[1], + trb.field[2], trb.field[3]); + xfer_len = trb.field[2] & 0xFFFFFF; + + if (cc == CC_SUCCESS || cc == CC_SHORT_PACKET) { + unsigned int i; + + /* Find matching intr endpoint and print HID data */ + for (i = 0; i < st->intr_count; i++) { + if (st->intr_eps[i].slot_id == slot && + st->intr_eps[i].dci == ep_id) { + u32 actual = st->intr_eps[i].max_packet - xfer_len; + + if (actual > 0 && actual <= INTR_BUF_SIZE) { + dev_info(&st->pdev->dev, + "[M%d] HID slot=%u ep=%u CC=%u len=%u: %*ph\n", + M_EVT_DELIV, + slot, ep_id, cc, actual, + min_t(u32, actual, 32), + st->intr_bufs[i]); + } + /* Requeue for next event */ + requeue_intr_trb(st, i); + break; + } + } + } else { + dev_warn(&st->pdev->dev, + "[M%d] Transfer event FAIL slot=%u ep=%u CC=%u\n", + M_EVT_DELIV, slot, ep_id, cc); + } + } else if (trb_type == TRB_TYPE_COMMAND_COMPLETION) { + dev_dbg(&st->pdev->dev, + "Command completion CC=%u slot=%u\n", cc, slot); + } else { + dev_dbg(&st->pdev->dev, + "Event type=%u CC=%u slot=%u ep=%u\n", + trb_type, cc, slot, ep_id); + } + } + + if (handled) { + ack_event(st); + return IRQ_HANDLED; + } + return IRQ_NONE; +} + +/* ----------------------------------------------------------------------- + * Controller init + * ----------------------------------------------------------------------- */ +static int xhci_init_controller(struct probe_state *st) +{ + u32 usbcmd, usbsts; + unsigned int i; + int ret; + + /* Pre-HCRST register dump: capture controller state before halt+reset */ + { + u32 pre_usbcmd = op_read32(st, USBCMD_OFF); + u32 pre_usbsts = op_read32(st, USBSTS_OFF); + u64 pre_dcbaap = op_read64(st, DCBAAP_OFF); + u64 pre_crcr = op_read64(st, CRCR_OFF); + u32 pre_config = op_read32(st, CONFIG_OFF); + + dev_info(&st->pdev->dev, "pre-HCRST state: USBCMD=0x%08x USBSTS=0x%08x DCBAAP=0x%016llx CRCR=0x%016llx CONFIG=0x%08x\n", + pre_usbcmd, pre_usbsts, (u64)pre_dcbaap, (u64)pre_crcr, pre_config); + dev_info(&st->pdev->dev, "pre-HCRST: RS=%u HCH=%u INTE=%u CNR=%u\n", + pre_usbcmd & 1, pre_usbsts & 1, (pre_usbcmd >> 2) & 1, (pre_usbsts >> 11) & 1); + } + + /* Halt controller if running */ + usbcmd = op_read32(st, USBCMD_OFF); + if (usbcmd & 1u) { + op_write32(st, USBCMD_OFF, usbcmd & ~1u); + for (i = 0; i < 100000; i++) { + if (op_read32(st, USBSTS_OFF) & 1u) + break; + udelay(1); + } + } + + /* Reset — enable MMIO write tracing from HCRST onward */ + mmio_trace_idx = 0; + mmio_trace_seq = 0; + mmio_trace_active = true; + op_write32(st, USBCMD_OFF, op_read32(st, USBCMD_OFF) | 2u); + for (i = 0; i < 100000; i++) { + if (!(op_read32(st, USBCMD_OFF) & 2u)) + break; + udelay(1); + } + /* Wait for CNR to clear */ + for (i = 0; i < 100000; i++) { + usbsts = op_read32(st, USBSTS_OFF); + if (!(usbsts & (1u << 11))) + break; + udelay(1); + } + + /* --- M2: CONTROLLER_RESET --- */ + ms_begin(st, M_RESET); + usbsts = op_read32(st, USBSTS_OFF); + ms_kv(st, M_RESET, "USBSTS=0x%08x HCH=%u CNR=%u", + usbsts, !!(usbsts & 1u), !!(usbsts & (1u << 11))); + ms_regs(st, M_RESET); + if ((usbsts & 1u) && !(usbsts & (1u << 11))) + ms_pass(st, M_RESET); + else + ms_fail(st, M_RESET, "HCH or CNR unexpected"); + + /* MaxSlotsEn */ + op_write32(st, CONFIG_OFF, st->max_slots); + op_write32(st, DNCTRL_OFF, 0x02); + + /* --- M3: DATA_STRUCTURES --- */ + ms_begin(st, M_DATA_STRUC); + ret = xhci_setup_rings(st); + if (ret) + return ret; + ms_regs(st, M_DATA_STRUC); + ms_pass(st, M_DATA_STRUC); + + /* Enable interrupter 0 */ + ir0_write32(st, IMAN_OFF, ir0_read32(st, IMAN_OFF) | 2u); + + /* Run + INTE */ + usbcmd = op_read32(st, USBCMD_OFF); + op_write32(st, USBCMD_OFF, usbcmd | 1u | (1u << 2)); + + /* --- M4: CONTROLLER_RUNNING --- */ + ms_begin(st, M_RUNNING); + { + u32 cmd = op_read32(st, USBCMD_OFF); + u32 sts = op_read32(st, USBSTS_OFF); + u32 iman = ir0_read32(st, IMAN_OFF); + + ms_kv(st, M_RUNNING, "USBCMD=0x%08x RS=%u INTE=%u", + cmd, !!(cmd & 1u), !!(cmd & (1u << 2))); + ms_kv(st, M_RUNNING, "USBSTS=0x%08x HCH=%u", sts, !!(sts & 1u)); + ms_kv(st, M_RUNNING, "IMAN=0x%08x IE=%u", iman, !!(iman & 2u)); + ms_regs(st, M_RUNNING); + if ((cmd & 1u) && (cmd & (1u << 2)) && (iman & 2u)) + ms_pass(st, M_RUNNING); + else + ms_fail(st, M_RUNNING, "RS/INTE/IE not set"); + } + return 0; +} + +/* ----------------------------------------------------------------------- + * PCI probe / remove + * ----------------------------------------------------------------------- */ +static int breenix_xhci_probe(struct pci_dev *pdev, + const struct pci_device_id *id) +{ + struct probe_state *st; + u32 cap_word; + u8 cap_length; + u32 hcsparams1; + bool port_enumerated[MAX_PORTS] = {}; + unsigned int i; + int ret; + + dev_info(&pdev->dev, "breenix_xhci_probe: claiming device\n"); + + /* DMA address diagnostics */ + { + void *test_ptr = kmalloc(64, GFP_KERNEL); + + if (test_ptr) { + dev_info(&pdev->dev, + "virt_to_phys test: kmalloc ptr %px -> phys 0x%llx\n", + test_ptr, (u64)virt_to_phys(test_ptr)); + kfree(test_ptr); + } + } + + dev_info(&pdev->dev, "BAR0 phys=0x%llx len=0x%lx\n", + (u64)pci_resource_start(pdev, 0), + (unsigned long)pci_resource_len(pdev, 0)); + +#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 5, 0) + dev_info(&pdev->dev, "IOMMU present: %s\n", + device_iommu_mapped(&pdev->dev) ? "YES" : "NO"); +#else + dev_info(&pdev->dev, "IOMMU present: %s (iommu_group check)\n", + pdev->dev.iommu_group ? "YES" : "NO"); +#endif + + st = kzalloc(sizeof(*st), GFP_KERNEL); + if (!st) + return -ENOMEM; + st->pdev = pdev; + pci_set_drvdata(pdev, st); + + ret = pci_enable_device(pdev); + if (ret) { + dev_err(&pdev->dev, "pci_enable_device failed: %d\n", ret); + goto err_free; + } + + pci_set_master(pdev); + + /* Dump PCI config space after pci_enable_device + pci_set_master */ + ms_pci_config(st, M_DISCOVERY, "pci_enabled"); + + ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64)); + if (ret) { + ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)); + if (ret) { + dev_err(&pdev->dev, "DMA mask setup failed\n"); + goto err_disable; + } + } + + st->bar_len = pci_resource_len(pdev, 0); + st->bar = pci_iomap(pdev, 0, st->bar_len); + if (!st->bar) { + dev_err(&pdev->dev, "pci_iomap BAR0 failed\n"); + ret = -EIO; + goto err_disable; + } + + /* Read capability registers */ + cap_word = readl(st->bar); + cap_length = cap_word & 0xFF; + st->hci_version = (cap_word >> 16) & 0xFFFF; + st->op_off = cap_length; + + hcsparams1 = readl(st->bar + 0x04); + st->max_slots = hcsparams1 & 0xFF; + st->max_ports = (hcsparams1 >> 24) & 0xFF; + + /* Context size: 64 if HCCPARAMS1 bit 2 set, else 32 */ + { + u32 hccparams1 = readl(st->bar + 0x10); + + st->ctx_size = (hccparams1 & (1u << 2)) ? 64 : 32; + } + + /* Runtime base: RTSOFF at cap_base + 0x18 */ + st->rt_off = readl(st->bar + 0x18) & ~0x1Fu; + /* Doorbell base: DBOFF at cap_base + 0x14 */ + st->db_off = readl(st->bar + 0x14) & ~0x3u; + + /* --- M1: CONTROLLER_DISCOVERY --- */ + ms_begin(st, M_DISCOVERY); + ms_kv(st, M_DISCOVERY, "BAR0_phys=0x%llx BAR0_len=0x%lx", + (u64)pci_resource_start(pdev, 0), + (unsigned long)pci_resource_len(pdev, 0)); + ms_kv(st, M_DISCOVERY, "xHCI_version=0x%04x", st->hci_version); + ms_kv(st, M_DISCOVERY, "max_slots=%u max_ports=%u ctx_size=%u", + st->max_slots, st->max_ports, st->ctx_size); + ms_kv(st, M_DISCOVERY, "cap_len=%u op_off=0x%x rt_off=0x%x db_off=0x%x", + cap_length, st->op_off, st->rt_off, st->db_off); + ms_kv(st, M_DISCOVERY, "HCSPARAMS1=0x%08x HCCPARAMS1=0x%08x", + hcsparams1, readl(st->bar + 0x10)); + ms_pass(st, M_DISCOVERY); + + /* Allocate control buffer */ + st->ctrl_buf = dma_alloc_coherent(&pdev->dev, CTRL_BUF_SIZE, + &st->ctrl_buf_dma, GFP_KERNEL); + if (!st->ctrl_buf) { + ret = -ENOMEM; + goto err_unmap; + } + ms_kv(st, M_DATA_STRUC, "ctrl_buf: virt=%px dma=0x%llx size=%u", + st->ctrl_buf, (u64)st->ctrl_buf_dma, CTRL_BUF_SIZE); + + /* Initialize controller (M2, M3, M4 happen inside) */ + ret = xhci_init_controller(st); + if (ret) { + dev_err(&pdev->dev, "xHCI init failed: %d\n", ret); + goto err_free_ctrl; + } + + /* Enumerate all ports */ + for (i = 0; i < st->max_ports; i++) + enumerate_port(st, i, port_enumerated); + + /* Wait for late connections, re-scan */ + msleep(2000); + for (i = 0; i < st->max_ports; i++) + enumerate_port(st, i, port_enumerated); + + dev_info(&pdev->dev, "Enumeration complete: %u interrupt endpoints\n", + st->intr_count); + + /* --- M10: INTERRUPT_TRANSFER --- */ + ms_begin(st, M_INTR_XFER); + ms_kv(st, M_INTR_XFER, "total_intr_eps=%u", st->intr_count); + + /* Allocate interrupt transfer buffers and queue TRBs */ + for (i = 0; i < st->intr_count; i++) { + struct intr_ep_queue *info = &st->intr_eps[i]; + struct xhci_trb trb; + u32 xfer_len; + + st->intr_bufs[i] = dma_alloc_coherent(&pdev->dev, INTR_BUF_SIZE, + &st->intr_bufs_dma[i], + GFP_KERNEL); + if (!st->intr_bufs[i]) + continue; + ms_kv(st, M_INTR_XFER, "intr_buf[%u]: virt=%px dma=0x%llx size=%u", + i, st->intr_bufs[i], (u64)st->intr_bufs_dma[i], + INTR_BUF_SIZE); + + xfer_len = info->max_packet; + if (xfer_len == 0) + xfer_len = 64; + if (xfer_len > INTR_BUF_SIZE) + xfer_len = INTR_BUF_SIZE; + + memset(st->intr_bufs[i], 0xDE, INTR_BUF_SIZE); + + memset(&trb, 0, sizeof(trb)); + trb.field[0] = lower_32_bits(st->intr_bufs_dma[i]); + trb.field[1] = upper_32_bits(st->intr_bufs_dma[i]); + trb.field[2] = xfer_len; + trb.field[3] = (TRB_TYPE_NORMAL << 10) | TRB_IOC | TRB_ISP; + + ms_trb(st, M_INTR_XFER, "intr_trb", &trb); + xhci_ring_enqueue_trb(info->ep_ring, &trb); + ring_doorbell(st, info->slot_id, info->dci); + ms_kv(st, M_INTR_XFER, "queued+doorbell: slot=%u dci=%u buf_dma=0x%llx len=%u", + info->slot_id, info->dci, + (u64)st->intr_bufs_dma[i], xfer_len); + { + struct xhci_virt_device *vd = + &st->virt_devs[info->slot_id - 1]; + struct xhci_ep_ctx *ep = + get_ep_ctx_out(vd->out_ctx, vd->ctx_size, + info->dci); + ms_kv(st, M_INTR_XFER, "ep_ctx_out: slot=%u dci=%u ep_info=0x%08x ep_info2=0x%08x deq=0x%016llx state=%u", + info->slot_id, info->dci, + ep->ep_info, ep->ep_info2, (u64)ep->deq, + ep->ep_info & 0x7); + } + } + + /* Dump MMIO write trace: HCRST through first doorbell ring */ + mmio_trace_active = false; + dev_info(&pdev->dev, "MMIO_TRACE: %u writes captured (max %u)\n", + mmio_trace_idx, MMIO_TRACE_MAX); + { + u32 t; + for (t = 0; t < mmio_trace_idx; t++) + dev_info(&pdev->dev, "MMIO_TRACE[%u] @%04x = %08x\n", + mmio_trace_buf[t].seq, + mmio_trace_buf[t].offset, + mmio_trace_buf[t].value); + } + + ms_regs(st, M_INTR_XFER); + ms_pass(st, M_INTR_XFER); + + /* Setup MSI interrupt */ + ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_MSI | PCI_IRQ_MSIX | + PCI_IRQ_INTX); + if (ret < 0) { + dev_err(&pdev->dev, "Failed to allocate IRQ vectors: %d\n", ret); + goto err_free_ctrl; + } + + /* Dump PCI config space after pci_alloc_irq_vectors (MSI configured) */ + ms_pci_config(st, M_DISCOVERY, "pci_msi"); + + st->irq = pci_irq_vector(pdev, 0); + ret = request_irq(st->irq, breenix_xhci_irq, IRQF_SHARED, + "breenix_xhci_probe", st); + if (ret) { + dev_err(&pdev->dev, "request_irq failed: %d\n", ret); + pci_free_irq_vectors(pdev); + goto err_free_ctrl; + } + + dev_info(&pdev->dev, + "Probe complete, IRQ=%d. Waiting for HID events...\n", + st->irq); + + /* --- M11: EVENT_DELIVERY --- polled check to verify event ring works */ + ms_begin(st, M_EVT_DELIV); + if (st->intr_count > 0) { + struct xhci_trb noop, noop_ev, ev; + + memset(&noop, 0, sizeof(noop)); + noop.field[3] = (TRB_TYPE_NOOP << 10); + if (xhci_cmd(st, &noop, &noop_ev) == 0) { + ms_kv(st, M_EVT_DELIV, "NOOP command: PASS (command ring alive)"); + ms_trb(st, M_EVT_DELIV, "noop_evt", &noop_ev); + } else { + ms_fail(st, M_EVT_DELIV, "NOOP command timeout (command ring dead)"); + } + + /* Quick poll for any transfer events */ + ms_kv(st, M_EVT_DELIV, "polling for transfer event (5s timeout)..."); + if (xhci_wait_for_event_ms(st, &ev, TRB_TYPE_TRANSFER_EVENT, + 5000) == 0) { + u32 cc = (ev.field[2] >> 24) & 0xFF; + u32 slot = (ev.field[3] >> TRB_SLOT_ID_SHIFT) & 0xFF; + u32 ep_id = (ev.field[3] >> TRB_EP_ID_SHIFT) & 0x1F; + + ms_kv(st, M_EVT_DELIV, "transfer_event: slot=%u ep=%u CC=%u", + slot, ep_id, cc); + ms_trb(st, M_EVT_DELIV, "xfer_evt", &ev); + ms_pass(st, M_EVT_DELIV); + } else { + ms_kv(st, M_EVT_DELIV, "no transfer events in 5s (try pressing a key)"); + } + } else { + ms_kv(st, M_EVT_DELIV, "no interrupt endpoints registered, skipping poll"); + } + + /* ----- Final state dump (all under M11) ----- */ + { + static const char * const ep_state_names[] = { + "disabled", "running", "halted", "stopped", + "error", "?5", "?6", "?7" + }; + + ms_regs(st, M_EVT_DELIV); + + /* Dump endpoint context state for each active interrupt EP */ + for (i = 0; i < st->intr_count; i++) { + struct intr_ep_queue *info = &st->intr_eps[i]; + struct xhci_virt_device *vdev; + struct xhci_ep_ctx *ep_ctx; + u32 ep_state; + + if (info->slot_id == 0 || info->slot_id > st->max_slots) + continue; + vdev = &st->virt_devs[info->slot_id - 1]; + if (!vdev->out_ctx) + continue; + + ep_ctx = get_ep_ctx_out(vdev->out_ctx, + vdev->ctx_size, info->dci); + ep_state = ep_ctx->ep_info & 0x7; + ms_kv(st, M_EVT_DELIV, "EP_STATE: slot=%u dci=%u ep_info=0x%08x state=%u(%s) deq=0x%016llx", + info->slot_id, info->dci, + ep_ctx->ep_info, ep_state, + ep_state_names[ep_state], + (u64)ep_ctx->deq); + } + + /* Dump doorbell register for each active slot */ + { + u8 seen_slots[MAX_SLOTS] = {}; + + for (i = 0; i < st->intr_count; i++) { + u8 sid = st->intr_eps[i].slot_id; + u32 db_val; + + if (sid == 0 || sid > st->max_slots) + continue; + if (seen_slots[sid - 1]) + continue; + seen_slots[sid - 1] = 1; + + db_val = xhci_read32(st, + st->db_off + (u32)sid * 4); + ms_kv(st, M_EVT_DELIV, "Doorbell[%u]=0x%08x", + sid, db_val); + } + } + + /* DCBAA dump */ + ms_kv(st, M_EVT_DELIV, "DCBAA entries:"); + for (i = 0; i <= st->max_slots && i <= MAX_SLOTS; i++) { + if (st->dcbaa[i]) + ms_kv(st, M_EVT_DELIV, "DCBAA[%u]=0x%016llx", + i, (u64)st->dcbaa[i]); + } + + /* Dump first TRB at current event ring dequeue pointer */ + if (st->event_dequeue) { + ms_trb(st, M_EVT_DELIV, "event_deq_trb", st->event_dequeue); + ms_kv(st, M_EVT_DELIV, "event_deq type=%u cycle=%u", + (st->event_dequeue->field[3] >> 10) & 0x3f, + st->event_dequeue->field[3] & 1); + } + } + + return 0; + +err_free_ctrl: + if (st->ctrl_buf) + dma_free_coherent(&pdev->dev, CTRL_BUF_SIZE, + st->ctrl_buf, st->ctrl_buf_dma); +err_unmap: + pci_iounmap(pdev, st->bar); +err_disable: + pci_disable_device(pdev); +err_free: + kfree(st); + return ret; +} + +static void breenix_xhci_remove(struct pci_dev *pdev) +{ + struct probe_state *st = pci_get_drvdata(pdev); + unsigned int i; + + dev_info(&pdev->dev, "breenix_xhci_remove\n"); + + /* Free IRQ */ + if (st->irq) { + free_irq(st->irq, st); + pci_free_irq_vectors(pdev); + } + + /* Halt controller */ + op_write32(st, USBCMD_OFF, op_read32(st, USBCMD_OFF) & ~1u); + msleep(10); + + /* Free interrupt buffers */ + for (i = 0; i < MAX_INTR_ENDPOINTS; i++) { + if (st->intr_bufs[i]) + dma_free_coherent(&pdev->dev, INTR_BUF_SIZE, + st->intr_bufs[i], + st->intr_bufs_dma[i]); + } + + /* Free device contexts */ + for (i = 0; i < MAX_SLOTS; i++) { + struct xhci_virt_device *vdev = &st->virt_devs[i]; + + if (vdev->in_ctx) + dma_free_coherent(&pdev->dev, 4096, + vdev->in_ctx, vdev->in_ctx_dma); + if (vdev->reconfig_in_ctx) + dma_free_coherent(&pdev->dev, 4096, + vdev->reconfig_in_ctx, + vdev->reconfig_in_ctx_dma); + if (vdev->out_ctx) + dma_free_coherent(&pdev->dev, 4096, + vdev->out_ctx, vdev->out_ctx_dma); + } + + /* Free segment TRB buffers */ + for (i = 0; i < st->seg_alloc_idx; i++) { + if (st->seg_trb_va[i]) + dma_free_coherent(&pdev->dev, + TRBS_PER_SEGMENT * sizeof(struct xhci_trb), + st->seg_trb_va[i], + st->seg_trb_dma[i]); + } + + /* Free ERST */ + if (st->erst) + dma_free_coherent(&pdev->dev, sizeof(struct xhci_erst_entry), + st->erst, st->erst_dma); + + /* Free DCBAA */ + if (st->dcbaa) + dma_free_coherent(&pdev->dev, 256 * sizeof(u64), + st->dcbaa, st->dcbaa_dma); + + /* Free ctrl buffer */ + if (st->ctrl_buf) + dma_free_coherent(&pdev->dev, CTRL_BUF_SIZE, + st->ctrl_buf, st->ctrl_buf_dma); + + pci_iounmap(pdev, st->bar); + pci_disable_device(pdev); + kfree(st); +} + +/* ----------------------------------------------------------------------- + * PCI device table — match xHCI controllers + * ----------------------------------------------------------------------- */ +static const struct pci_device_id breenix_xhci_ids[] = { + /* NEC/Renesas uPD720200 (common in Parallels) */ + { PCI_DEVICE(0x1033, 0x0194) }, + /* Intel xHCI */ + { PCI_DEVICE_CLASS(PCI_CLASS_SERIAL_USB_XHCI, ~0) }, + { 0 } +}; +MODULE_DEVICE_TABLE(pci, breenix_xhci_ids); + +static struct pci_driver breenix_xhci_driver = { + .name = "breenix_xhci_probe", + .id_table = breenix_xhci_ids, + .probe = breenix_xhci_probe, + .remove = breenix_xhci_remove, +}; + +module_pci_driver(breenix_xhci_driver); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Ryan Breen "); +MODULE_DESCRIPTION("Standalone xHCI probe for Breenix validation"); diff --git a/linux_xhci_module/deploy.sh b/linux_xhci_module/deploy.sh new file mode 100755 index 00000000..84ce9ad3 --- /dev/null +++ b/linux_xhci_module/deploy.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# Deploy and load breenix_xhci_probe module on linux-probe Parallels VM. +# +# Usage: bash deploy.sh [--tail] +# --tail: Keep tailing dmesg after loading (Ctrl-C to stop) +# +set -euo pipefail + +VM=linux-probe +SRC_DIR="$(cd "$(dirname "$0")" && pwd)" +TAIL=false + +for arg in "$@"; do + case "$arg" in + --tail) TAIL=true ;; + *) echo "Unknown arg: $arg"; exit 1 ;; + esac +done + +echo "=== Ensuring VM is running ===" +prlctl resume "$VM" 2>/dev/null || prlctl start "$VM" 2>/dev/null || true +# Wait for VM to be reachable +for i in $(seq 1 30); do + if prlctl exec "$VM" -- true 2>/dev/null; then + break + fi + echo " Waiting for VM... ($i/30)" + sleep 2 +done + +echo "=== Installing build dependencies ===" +prlctl exec "$VM" -- apk add --no-cache build-base linux-headers 2>/dev/null || true + +echo "=== Copying module source ===" +prlctl exec "$VM" -- mkdir -p /root/xhci_module +for f in Makefile breenix_xhci_probe.c; do + prlctl exec "$VM" -- sh -c "cat > /root/xhci_module/$f" < "$SRC_DIR/$f" +done + +echo "=== Building module ===" +prlctl exec "$VM" -- sh -c "cd /root/xhci_module && make clean && make" 2>&1 + +echo "=== Finding xHCI PCI address ===" +XHCI_LINE=$(prlctl exec "$VM" -- lspci -nn 2>/dev/null | grep -i xhci | head -1 || true) +if [ -z "$XHCI_LINE" ]; then + echo "ERROR: No xHCI controller found in lspci output" + echo "Available PCI devices:" + prlctl exec "$VM" -- lspci -nn + exit 1 +fi +XHCI_ADDR=$(echo "$XHCI_LINE" | awk '{print $1}') +echo "xHCI at PCI $XHCI_ADDR: $XHCI_LINE" + +echo "=== Unbinding stock xhci_hcd driver ===" +prlctl exec "$VM" -- sh -c "echo 0000:$XHCI_ADDR > /sys/bus/pci/drivers/xhci_hcd/unbind" 2>/dev/null || true + +echo "=== Unloading previous module version ===" +prlctl exec "$VM" -- rmmod breenix_xhci_probe 2>/dev/null || true + +echo "=== Loading module ===" +prlctl exec "$VM" -- insmod /root/xhci_module/breenix_xhci_probe.ko + +echo "" +echo "=== Module loaded successfully ===" +echo "=== Recent dmesg output: ===" +prlctl exec "$VM" -- dmesg | tail -80 + +if [ "$TAIL" = true ]; then + echo "" + echo "=== Tailing dmesg (Ctrl-C to stop) ===" + prlctl exec "$VM" -- dmesg -w +fi diff --git a/parallels-loader/src/hw_config.rs b/parallels-loader/src/hw_config.rs index e15874bc..002ec446 100644 --- a/parallels-loader/src/hw_config.rs +++ b/parallels-loader/src/hw_config.rs @@ -109,6 +109,15 @@ pub struct HardwareConfig { // --- Timer --- /// Generic timer frequency in Hz (from CNTFRQ_EL0) pub timer_freq_hz: u64, + + // --- xHCI --- + /// Set to 1 if the loader already performed HCRST before ExitBootServices. + /// The kernel should skip HCRST and just reconfigure rings. + pub xhci_hcrst_done: u32, + pub _pad6: u32, + /// xHCI BAR0 physical address (read from PCI ECAM by the loader). + /// 0 if not discovered. + pub xhci_bar_phys: u64, } pub const HARDWARE_CONFIG_MAGIC: u32 = 0x4252_4E58; // "BRNX" @@ -152,6 +161,9 @@ impl HardwareConfig { }, rsdp_addr: 0, timer_freq_hz: 0, + xhci_hcrst_done: 0, + _pad6: 0, + xhci_bar_phys: 0, } } diff --git a/parallels-loader/src/kernel_entry.rs b/parallels-loader/src/kernel_entry.rs index 3474aaab..5b2ea69b 100644 --- a/parallels-loader/src/kernel_entry.rs +++ b/parallels-loader/src/kernel_entry.rs @@ -74,7 +74,8 @@ unsafe fn switch_and_jump(ttbr0: u64, ttbr1: u64, entry: u64, hw_config_ptr: u64 // Must match kernel boot.S layout: // Index 0: Device-nGnRnE (0x00) // Index 1: Normal WB (0xFF) - "mov x4, #0xFF00", + // Index 2: Normal NC (0x44) — for DMA buffers + "ldr x4, ={mair}", "msr mair_el1, x4", "isb", @@ -137,6 +138,7 @@ unsafe fn switch_and_jump(ttbr0: u64, ttbr1: u64, entry: u64, hw_config_ptr: u64 "mov x0, x3", // x0 = hw_config_ptr "br x2", // Jump to kernel_main + mair = const page_tables::MAIR_VALUE, tcr = const page_tables::TCR_VALUE, in("x0") ttbr0, in("x1") ttbr1, diff --git a/parallels-loader/src/main.rs b/parallels-loader/src/main.rs index 7ae021f1..74ba86a0 100644 --- a/parallels-loader/src/main.rs +++ b/parallels-loader/src/main.rs @@ -130,14 +130,33 @@ fn main() -> Status { } } + // --- xHCI: DisconnectController DISABLED --- + // + // EXPERIMENT: DisconnectController was destroying UEFI's endpoint state + // (sending ConfigureEndpoint-Deconfigure commands) before ExitBootServices. + // This caused the Parallels hypervisor to show NO "DisableEndpoint while + // io_cnt is not zero!" during HCRST, and NO ep creates after HCRST → CC=12. + // + // On linux-probe (where CC=12 does NOT occur), the Linux EFI stub does NOT + // call DisconnectController. UEFI's endpoints survive through ExitBootServices + // to the linux module's HCRST, where "DisableEndpoint while io_cnt is not zero!" + // appears, and subsequent ConfigureEndpoint commands DO produce ep creates. + // + // By SKIPPING DisconnectController, we let UEFI's in-flight USB I/O persist + // through ExitBootServices, matching linux-probe's behavior. + config.xhci_hcrst_done = 0; + log::info!("xHCI DisconnectController SKIPPED (matching linux-probe behavior)"); + log::info!("--- Exiting Boot Services ---"); // Exit UEFI boot services. After this, NO UEFI calls are possible. - // The memory map is required for ExitBootServices. unsafe { let _ = uefi::boot::exit_boot_services(MemoryType::LOADER_DATA); } + // Post-EBS BAR re-enable DISABLED — let the kernel find the device in + // whatever state EBS leaves it (matches linux-probe where Command=0x0010). + // Jump to kernel with our page tables and HardwareConfig let page_tables = unsafe { &mut *(&raw mut PAGE_TABLES) }; let hw_config = unsafe { &*(&raw const HW_CONFIG) }; @@ -240,3 +259,222 @@ fn halt() -> ! { unsafe { core::arch::asm!("wfi") }; } } + +// --------------------------------------------------------------------------- +// Minimal EFI_PCI_IO_PROTOCOL binding for disconnect_xhci_driver() +// --------------------------------------------------------------------------- + +/// GetLocation function pointer type. +type PciIoGetLocationFn = unsafe extern "efiapi" fn( + this: *const PciIoProtocol, + segment: *mut usize, + bus: *mut usize, + device: *mut usize, + function: *mut usize, +) -> Status; + +/// Minimal EFI_PCI_IO_PROTOCOL — only need GetLocation for identifying the device. +/// Layout must match UEFI Spec 2.10 Section 14.4 up through GetLocation. +#[repr(C)] +#[uefi::proto::unsafe_protocol("4cf5b200-68b8-4ca5-9eec-b23e3f50029a")] +struct PciIoProtocol { + poll_mem: usize, + poll_io: usize, + mem_read: usize, + mem_write: usize, + io_read: usize, + io_write: usize, + pci_read: usize, + pci_write: usize, + copy_mem: usize, + map: usize, + unmap: usize, + allocate_buffer: usize, + free_buffer: usize, + flush: usize, + get_location: PciIoGetLocationFn, +} + +/// Disconnect UEFI's xHCI driver from the PCI device at 00:03.0. +/// +/// Returns a sentinel value for diagnostics: +/// 0x01 = success (driver disconnected) +/// 0xA1 = no PCI handles found +/// 0xA2 = xHCI device not found among handles +/// 0xA3 = disconnect_controller failed +fn disconnect_xhci_driver() -> u32 { + use uefi::boot; + + // Find all handles with EFI_PCI_IO_PROTOCOL + let handles = match boot::locate_handle_buffer( + boot::SearchType::ByProtocol(&::GUID), + ) { + Ok(h) => h, + Err(_) => { + log::warn!("xHCI disconnect: no PCI handles found"); + return 0xA1; + } + }; + + log::info!("xHCI disconnect: found {} PCI device handles", handles.len()); + + let image = boot::image_handle(); + + for &handle in handles.iter() { + // Open protocol just to peek at GetLocation + let pci_io = match unsafe { + boot::open_protocol::( + boot::OpenProtocolParams { + handle, + agent: image, + controller: None, + }, + boot::OpenProtocolAttributes::GetProtocol, + ) + } { + Ok(p) => p, + Err(_) => continue, + }; + + // Check if this is 00:03.0 (the xHCI device) + let (mut seg, mut bus, mut dev, mut fun) = (0usize, 0, 0, 0); + let status = unsafe { + (pci_io.get_location)( + &*pci_io as *const PciIoProtocol, + &mut seg, + &mut bus, + &mut dev, + &mut fun, + ) + }; + + if status != Status::SUCCESS { + continue; + } + + if bus != 0 || dev != 3 || fun != 0 { + continue; + } + + log::info!("xHCI disconnect: found device at {:04x}:{:02x}:{:02x}.{:x}", seg, bus, dev, fun); + + // Drop the protocol reference before disconnecting + drop(pci_io); + + // Disconnect ALL drivers from this controller + match boot::disconnect_controller(handle, None, None) { + Ok(_) => { + log::info!("xHCI disconnect: SUCCESS — UEFI driver detached"); + return 0x01; + } + Err(e) => { + log::warn!("xHCI disconnect: disconnect_controller failed: {:?}", e); + return 0xA3; + } + } + } + + log::warn!("xHCI disconnect: device 00:03.0 not found"); + 0xA2 +} + +/// Halt and reset the xHCI controller BEFORE ExitBootServices. +/// +/// The Parallels hypervisor tracks internal USB endpoint state. If we reset +/// the xHCI after EBS, the endpoint teardown fails ("DisableEndpoint while +/// io_cnt is not zero") and subsequent ConfigureEndpoint commands don't +/// create new internal endpoints. Doing halt+HCRST before EBS, while UEFI +/// services are alive, lets the hypervisor cleanly shut down endpoints. +fn pre_ebs_xhci_halt_reset(ecam_base: u64) { + if ecam_base == 0 { + log::warn!("xHCI pre-EBS reset: no ECAM base, skipping"); + return; + } + + // Read BAR0 from PCI config space (ECAM offset for 00:03.0 = 0x18000) + let ecam_xhci = ecam_base + 0x18000; + let bar0 = unsafe { core::ptr::read_volatile((ecam_xhci + 0x10) as *const u32) }; + let bar0_phys = (bar0 & 0xFFFFF000) as u64; + + if bar0_phys == 0 { + log::warn!("xHCI pre-EBS reset: BAR0 is zero, skipping"); + return; + } + + log::info!("xHCI pre-EBS reset: BAR0=0x{:08x}", bar0_phys); + + // Read cap_length to find operational registers + let cap_word = unsafe { core::ptr::read_volatile(bar0_phys as *const u32) }; + let cap_length = (cap_word & 0xFF) as u64; + let op_base = bar0_phys + cap_length; + + log::info!("xHCI pre-EBS reset: cap_length={} op_base=0x{:x}", cap_length, op_base); + + // Read current USBCMD and USBSTS + let usbcmd = unsafe { core::ptr::read_volatile(op_base as *const u32) }; + let usbsts = unsafe { core::ptr::read_volatile((op_base + 4) as *const u32) }; + log::info!("xHCI pre-EBS: USBCMD=0x{:08x} USBSTS=0x{:08x}", usbcmd, usbsts); + + // Step 1: Halt the controller (clear RS bit 0) + if usbcmd & 1 != 0 { + unsafe { + core::ptr::write_volatile(op_base as *mut u32, usbcmd & !1); + } + // Wait for HCH (USBSTS bit 0) — up to 16ms per xHCI spec + let mut halted = false; + for _ in 0..1_000_000 { + let sts = unsafe { core::ptr::read_volatile((op_base + 4) as *const u32) }; + if sts & 1 != 0 { + halted = true; + break; + } + } + if halted { + log::info!("xHCI pre-EBS: controller halted (HCH=1)"); + } else { + log::warn!("xHCI pre-EBS: halt timeout, proceeding with HCRST anyway"); + } + } else { + log::info!("xHCI pre-EBS: controller already halted (RS=0)"); + } + + // Step 2: HCRST (set bit 1 of USBCMD) + let usbcmd_now = unsafe { core::ptr::read_volatile(op_base as *const u32) }; + unsafe { + core::ptr::write_volatile(op_base as *mut u32, usbcmd_now | (1 << 1)); + } + + // Wait for HCRST bit to self-clear (up to 16ms per spec) + let mut reset_done = false; + for _ in 0..1_000_000 { + let cmd = unsafe { core::ptr::read_volatile(op_base as *const u32) }; + if cmd & (1 << 1) == 0 { + reset_done = true; + break; + } + } + + if reset_done { + // Wait for CNR (Controller Not Ready, USBSTS bit 11) to clear + let mut ready = false; + for _ in 0..1_000_000 { + let sts = unsafe { core::ptr::read_volatile((op_base + 4) as *const u32) }; + if sts & (1 << 11) == 0 { + ready = true; + break; + } + } + if ready { + log::info!("xHCI pre-EBS: HCRST complete, controller ready"); + } else { + log::warn!("xHCI pre-EBS: CNR still set after HCRST"); + } + } else { + log::warn!("xHCI pre-EBS: HCRST timeout"); + } + + // Read final state + let usbcmd_final = unsafe { core::ptr::read_volatile(op_base as *const u32) }; + let usbsts_final = unsafe { core::ptr::read_volatile((op_base + 4) as *const u32) }; + log::info!("xHCI pre-EBS: final USBCMD=0x{:08x} USBSTS=0x{:08x}", usbcmd_final, usbsts_final); +} diff --git a/parallels-loader/src/page_tables.rs b/parallels-loader/src/page_tables.rs index ea3a2eac..5e288bb1 100644 --- a/parallels-loader/src/page_tables.rs +++ b/parallels-loader/src/page_tables.rs @@ -27,8 +27,10 @@ mod attr { /// MUST match kernel boot.S MAIR layout: /// Index 0 = Device-nGnRnE (0x00) /// Index 1 = Normal WB-WA (0xFF) + /// Index 2 = Normal NC (0x44) pub const ATTR_IDX_DEVICE: u64 = 0 << 2; // MAIR index 0: Device-nGnRnE pub const ATTR_IDX_NORMAL: u64 = 1 << 2; // MAIR index 1: Normal WB + pub const ATTR_IDX_NC: u64 = 2 << 2; // MAIR index 2: Normal Non-Cacheable /// Access flag (must be set, or access fault) pub const AF: u64 = 1 << 10; @@ -48,6 +50,9 @@ mod attr { /// Normal memory block: cacheable, inner shareable pub const NORMAL_BLOCK: u64 = VALID | BLOCK | ATTR_IDX_NORMAL | AF | ISH | AP_RW_EL1; + /// Normal Non-Cacheable block: for DMA buffers (no cache coherency needed) + pub const NC_BLOCK: u64 = VALID | BLOCK | ATTR_IDX_NC | AF | ISH | AP_RW_EL1; + /// Device memory block: non-cacheable, outer shareable, execute-never pub const DEVICE_BLOCK: u64 = VALID | BLOCK | ATTR_IDX_DEVICE | AF | OSH | AP_RW_EL1 | UXN | PXN; @@ -59,7 +64,8 @@ mod attr { /// MUST match kernel boot.S layout: /// Index 0: Device-nGnRnE (0x00) /// Index 1: Normal WB cacheable (0xFF = Inner WB RA WA, Outer WB RA WA) -pub const MAIR_VALUE: u64 = 0x00_00_00_00_00_00_FF00; +/// Index 2: Normal Non-Cacheable (0x44 = Inner NC, Outer NC) — for DMA buffers +pub const MAIR_VALUE: u64 = 0x00_00_00_00_0044_FF00; /// TCR (Translation Control Register) value for 4K granule, 48-bit VA. /// T0SZ = 16 (48-bit VA for TTBR0) @@ -86,13 +92,24 @@ pub const TCR_VALUE: u64 = (16 << 0) // T0SZ /// Size of a single page table (4KB, 512 entries of 8 bytes each). const PAGE_TABLE_SIZE: usize = 4096; +/// Physical base of the 2MB Non-Cacheable DMA region. +/// xHCI (and future DMA drivers) allocate buffers from this region +/// to avoid cache coherency issues on non-coherent ARM64 platforms. +/// Placed above the kernel heap range (0x42000000-0x50000000) so the +/// general-purpose allocator never touches this memory. +pub const NC_DMA_BASE: u64 = 0x5000_0000; + +/// Size of the NC DMA region (2MB). +pub const NC_DMA_SIZE: u64 = 0x20_0000; + /// Number of page tables we pre-allocate. /// L0 (TTBR0): 1 /// L0 (TTBR1): 1 /// L1 (TTBR0): 1 (covers 512GB) /// L1 (TTBR1): 1 (covers 512GB) -/// L2 (for device regions): 2 (for 0x00000000-0x3FFFFFFF, 0x10000000-0x1FFFFFFF) -const MAX_PAGE_TABLES: usize = 8; +/// L2 (for device regions): 2 (for 0x00000000-0x3FFFFFFF) +/// L2 (for RAM with NC block): 2 (for 0x40000000-0x7FFFFFFF) +const MAX_PAGE_TABLES: usize = 10; /// Page table storage. Allocated in the loader's BSS. /// Must be 4KB aligned. @@ -170,10 +187,28 @@ pub fn build_page_tables(storage: &mut PageTableStorage) -> (u64, u64) { } // --- RAM: 0x40000000 - 0xBFFFFFFF (2GB, L1 entries 1-2) --- - // Use 1GB block mappings for RAM (much simpler, fewer TLB entries) - // L1[1] = 0x40000000 - 0x7FFFFFFF (1GB block, normal memory) - write_entry(ttbr0_l1, 1, 0x4000_0000 | attr::NORMAL_BLOCK); - write_entry(ttbr1_l1, 1, 0x4000_0000 | attr::NORMAL_BLOCK); + // + // L1[1] = 0x40000000 - 0x7FFFFFFF: Use L2 table to carve out a 2MB NC block + // for DMA buffers at NC_DMA_BASE. All other 2MB blocks are WB-WA. + let ttbr0_l2_ram = storage.alloc_table(); + let ttbr1_l2_ram = storage.alloc_table(); + write_entry(ttbr0_l1, 1, ttbr0_l2_ram | attr::TABLE_DESC); + write_entry(ttbr1_l1, 1, ttbr1_l2_ram | attr::TABLE_DESC); + + // NC block index: (NC_DMA_BASE - 0x40000000) / 0x200000 + // DMA buffers are mapped Non-Cacheable for hardware DMA coherency. + // Tested: switching to Normal Cacheable (WB) does NOT fix CC=12 on Parallels. + let nc_block_idx = ((NC_DMA_BASE - 0x4000_0000) / 0x20_0000) as usize; + for i in 0..512u64 { + let phys = 0x4000_0000 + i * 0x20_0000; + let block_attr = if i as usize == nc_block_idx { + attr::NC_BLOCK + } else { + attr::NORMAL_BLOCK + }; + write_entry(ttbr0_l2_ram, i as usize, phys | block_attr); + write_entry(ttbr1_l2_ram, i as usize, phys | block_attr); + } // L1[2] = 0x80000000 - 0xBFFFFFFF (1GB block, normal memory) write_entry(ttbr0_l1, 2, 0x8000_0000 | attr::NORMAL_BLOCK); diff --git a/scripts/parallels/deploy-to-vm.sh b/scripts/parallels/deploy-to-vm.sh deleted file mode 100755 index 8aa6f42e..00000000 --- a/scripts/parallels/deploy-to-vm.sh +++ /dev/null @@ -1,213 +0,0 @@ -#!/bin/bash -# Deploy Breenix to the Parallels Desktop VM. -# -# This script packages the already-built kernel + UEFI loader into a -# Parallels .hdd disk image, reconfigures the VM, and optionally starts it. -# It does NOT build anything — run build-efi.sh --kernel first. -# -# Usage: -# ./scripts/parallels/deploy-to-vm.sh # Deploy only (don't start) -# ./scripts/parallels/deploy-to-vm.sh --boot # Deploy and start VM -# -# Typical workflow: -# touch kernel/src/drivers/usb/xhci.rs -# scripts/parallels/build-efi.sh --kernel -# scripts/parallels/deploy-to-vm.sh --boot -# sleep 50 -# cat /tmp/breenix-parallels-serial.log | grep "BUILD_ID" - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -PARALLELS_DIR="$PROJECT_ROOT/target/parallels" -SERIAL_LOG="/tmp/breenix-parallels-serial.log" -HDD_DIR="$PARALLELS_DIR/breenix-efi.hdd" -EXT2_HDD_DIR="$PARALLELS_DIR/breenix-ext2.hdd" -EXT2_DISK="$PROJECT_ROOT/target/ext2-aarch64.img" -PARALLELS_VM="breenix-dev" -BOOT=false - -for arg in "$@"; do - case "$arg" in - --boot) BOOT=true ;; - --vm) PARALLELS_VM="$2"; shift ;; - *) echo "Unknown argument: $arg"; exit 1 ;; - esac -done - -# Check for required tools -if ! command -v prlctl &>/dev/null; then - echo "ERROR: prlctl not found. Is Parallels Desktop installed?" - exit 1 -fi -if ! command -v prl_disk_tool &>/dev/null; then - echo "ERROR: prl_disk_tool not found. Is Parallels Desktop installed?" - exit 1 -fi - -# Verify built artifacts exist -LOADER_EFI="$PROJECT_ROOT/target/aarch64-unknown-uefi/release/parallels-loader.efi" -KERNEL_ELF="$PROJECT_ROOT/target/aarch64-breenix/release/kernel-aarch64" - -if [ ! -f "$LOADER_EFI" ]; then - echo "ERROR: UEFI loader not found at $LOADER_EFI" - echo "Run: cargo build --release --target aarch64-unknown-uefi -p parallels-loader" - exit 1 -fi -if [ ! -f "$KERNEL_ELF" ]; then - echo "ERROR: Kernel not found at $KERNEL_ELF" - echo "Run: scripts/parallels/build-efi.sh --kernel" - exit 1 -fi - -LOADER_SIZE=$(stat -f%z "$LOADER_EFI" 2>/dev/null || stat -c%s "$LOADER_EFI") -KERNEL_SIZE=$(stat -f%z "$KERNEL_ELF" 2>/dev/null || stat -c%s "$KERNEL_ELF") -KERNEL_MTIME=$(stat -f "%Sm" "$KERNEL_ELF" 2>/dev/null || stat -c "%y" "$KERNEL_ELF") - -echo "=== Packaging Parallels EFI disk ===" -echo " Loader: $LOADER_SIZE bytes" -echo " Kernel: $KERNEL_SIZE bytes (built $KERNEL_MTIME)" - -# Check if kernel binary contains a BUILD_ID (diagnostic) -if command -v strings &>/dev/null; then - BUILD_ID_IN_BINARY=$(strings "$KERNEL_ELF" | grep "BUILD_ID:" | head -1 || true) - if [ -n "$BUILD_ID_IN_BINARY" ]; then - echo " $BUILD_ID_IN_BINARY" - else - echo " WARNING: No BUILD_ID found in kernel binary (stale build?)" - fi -fi - -mkdir -p "$PARALLELS_DIR" - -# Create GPT+FAT32 disk image using hdiutil (native macOS) -DMG_PATH="$PARALLELS_DIR/efi-temp.dmg" -rm -f "$DMG_PATH" -hdiutil create -size 64m -fs FAT32 -volname BREENIX -layout GPTSPUD "$DMG_PATH" >/dev/null 2>&1 - -VOLUME=$(hdiutil attach "$DMG_PATH" 2>/dev/null | grep -o '/Volumes/[^ ]*' | head -1) -if [ -z "$VOLUME" ] || [ ! -d "$VOLUME" ]; then - echo "ERROR: Failed to mount FAT32 disk image" - rm -f "$DMG_PATH" - exit 1 -fi - -mkdir -p "$VOLUME/EFI/BOOT" -mkdir -p "$VOLUME/EFI/BREENIX" -cp "$LOADER_EFI" "$VOLUME/EFI/BOOT/BOOTAA64.EFI" -cp "$KERNEL_ELF" "$VOLUME/EFI/BREENIX/KERNEL" -hdiutil detach "$VOLUME" >/dev/null 2>&1 - -# Convert DMG to raw disk image -RAW_IMG="$PARALLELS_DIR/efi-raw.img" -rm -f "$RAW_IMG" "${RAW_IMG}.cdr" -hdiutil convert "$DMG_PATH" -format UDTO -o "$RAW_IMG" >/dev/null 2>&1 -mv "${RAW_IMG}.cdr" "$RAW_IMG" -rm -f "$DMG_PATH" - -# Patch GPT partition type from "Microsoft Basic Data" to "EFI System Partition" -# so UEFI firmware recognizes the ESP and auto-boots BOOTAA64.EFI -python3 "$PROJECT_ROOT/scripts/parallels/patch-gpt-esp.py" "$RAW_IMG" - -# Wrap EFI disk in Parallels .hdd format -rm -rf "$HDD_DIR" -prl_disk_tool create --hdd "$HDD_DIR" --size 64M >/dev/null 2>&1 -HDS_FILE=$(find "$HDD_DIR" -name "*.hds" | head -1) -if [ -z "$HDS_FILE" ]; then - echo "ERROR: No .hds file found in $HDD_DIR" - rm -f "$RAW_IMG" - exit 1 -fi -cp "$RAW_IMG" "$HDS_FILE" -rm -f "$RAW_IMG" -echo " EFI disk: $HDD_DIR" - -# Wrap ext2 data disk if available -if [ -f "$EXT2_DISK" ]; then - EXT2_SIZE_MB=$(( ($(stat -f%z "$EXT2_DISK" 2>/dev/null || stat -c%s "$EXT2_DISK") + 1048575) / 1048576 )) - rm -rf "$EXT2_HDD_DIR" - prl_disk_tool create --hdd "$EXT2_HDD_DIR" --size "${EXT2_SIZE_MB}M" >/dev/null 2>&1 - EXT2_HDS=$(find "$EXT2_HDD_DIR" -name "*.hds" | head -1) - if [ -n "$EXT2_HDS" ]; then - cp "$EXT2_DISK" "$EXT2_HDS" - echo " ext2 disk: $EXT2_HDD_DIR (${EXT2_SIZE_MB}MB)" - fi -fi - -echo "" -echo "=== Configuring Parallels VM '$PARALLELS_VM' ===" - -# Create VM if it doesn't exist -if ! prlctl list --all 2>/dev/null | grep -q "$PARALLELS_VM"; then - echo "Creating VM '$PARALLELS_VM'..." - prlctl create "$PARALLELS_VM" --ostype linux --distribution linux --no-hdd - prlctl set "$PARALLELS_VM" --memsize 2048 - prlctl set "$PARALLELS_VM" --cpus 4 -fi - -# Force-stop the VM -echo "Stopping VM (force kill)..." -prlctl stop "$PARALLELS_VM" --kill 2>/dev/null || true - -# Poll until confirmed stopped -for i in $(seq 1 20); do - VM_STATUS=$(prlctl status "$PARALLELS_VM" 2>/dev/null | awk '{print $NF}') - if [ "$VM_STATUS" = "stopped" ]; then - echo "VM is stopped." - break - fi - if [ "$i" -eq 20 ]; then - echo "WARNING: VM did not stop after 20s. Proceeding anyway." - echo "If stuck, run: sudo pkill -9 -f prl_disp_service" - fi - sleep 1 -done - -# Configure VM: EFI boot, remove all SATA devices, attach our disks -prlctl set "$PARALLELS_VM" --efi-boot on 2>/dev/null || true - -for dev in hdd0 hdd1 hdd2 hdd3 cdrom0 cdrom1; do - prlctl set "$PARALLELS_VM" --device-del "$dev" 2>/dev/null || true -done - -prlctl set "$PARALLELS_VM" --device-add hdd --image "$HDD_DIR" --type plain --position 0 -if [ -d "$EXT2_HDD_DIR" ]; then - prlctl set "$PARALLELS_VM" --device-add hdd --image "$EXT2_HDD_DIR" --type plain --position 1 - echo " hdd0: EFI boot disk (FAT32) at sata:0" - echo " hdd1: ext2 data disk at sata:1" -else - echo " hdd0: EFI boot disk (FAT32) at sata:0" -fi - -prlctl set "$PARALLELS_VM" --device-bootorder "hdd0" 2>/dev/null || true - -# Configure serial port output to file -prlctl set "$PARALLELS_VM" --device-del serial0 2>/dev/null || true -prlctl set "$PARALLELS_VM" --device-add serial --output "$SERIAL_LOG" 2>/dev/null || true -prlctl set "$PARALLELS_VM" --device-set serial0 --connect 2>/dev/null || true - -# Delete NVRAM to ensure fresh UEFI boot state -VM_DIR="$HOME/Parallels/${PARALLELS_VM}.pvm" -if [ -f "$VM_DIR/NVRAM.dat" ]; then - rm -f "$VM_DIR/NVRAM.dat" - echo " NVRAM deleted (fresh UEFI state)" -fi - -# Truncate serial log -> "$SERIAL_LOG" -echo " Serial log truncated: $SERIAL_LOG" - -if [ "$BOOT" = true ]; then - echo "" - echo "=== Starting VM ===" - prlctl start "$PARALLELS_VM" - echo "VM started." - echo "" - echo "Monitor: tail -f $SERIAL_LOG" - echo "Stop: prlctl stop $PARALLELS_VM --kill" -fi - -echo "" -echo "Deploy complete." diff --git a/userspace/programs/Cargo.toml b/userspace/programs/Cargo.toml index 7e756f58..8e497ea4 100644 --- a/userspace/programs/Cargo.toml +++ b/userspace/programs/Cargo.toml @@ -480,6 +480,10 @@ path = "src/btop.rs" name = "burl" path = "src/burl.rs" +[[bin]] +name = "btrace" +path = "src/btrace.rs" + [[bin]] name = "blogd" path = "src/blogd.rs" diff --git a/userspace/programs/build.sh b/userspace/programs/build.sh index 8506a3f8..b5eb4eaf 100755 --- a/userspace/programs/build.sh +++ b/userspace/programs/build.sh @@ -252,6 +252,9 @@ STD_BINARIES=( "telnetd" "blogd" + # Diagnostics + "btrace" + # Log viewer "bless" diff --git a/userspace/programs/src/btrace.rs b/userspace/programs/src/btrace.rs new file mode 100644 index 00000000..9dc81bd7 --- /dev/null +++ b/userspace/programs/src/btrace.rs @@ -0,0 +1,338 @@ +//! btrace — Breenix Trace Reader +//! +//! Reads kernel and xHCI trace data from procfs and prints a compact +//! diagnostic summary to stdout (which goes to serial). Runs once, +//! suitable for being called periodically by init or from the shell. +//! +//! Reads: +//! /proc/uptime — system uptime +//! /proc/stat — syscalls, interrupts, context switches, etc. +//! /proc/xhci/trace — xHCI hardware trace buffer +//! +//! Output format: +//! === btrace @ 42.5s === +//! sys=1234 irq=5678 ctx=901 fork=12 exec=8 cow=3 +//! xhci: 156 records, last_cc=12 (EP_NOT_ENABLED) slot=1 ep=3 +//! xhci: 45 CMD_COMPLETE 89 XFER_EVENT 22 NOTE +//! ===================== + +use libbreenix::fs::{self, O_RDONLY}; +use libbreenix::io; + +/// Read the full contents of a procfs file into a Vec. +fn read_procfs(path: &str) -> Vec { + match fs::open(path, O_RDONLY) { + Ok(fd) => { + let mut buf = vec![0u8; 262144]; + let mut total = 0; + loop { + match io::read(fd, &mut buf[total..]) { + Ok(n) if n > 0 => { + total += n; + if total >= buf.len() - 256 { + buf.resize(buf.len() + 4096, 0); + } + } + _ => break, + } + } + let _ = io::close(fd); + buf.truncate(total); + buf + } + Err(_) => Vec::new(), + } +} + +/// Parse a "key value" line from /proc/stat, returning the value as u64. +fn parse_stat_value(stat: &str, key: &str) -> u64 { + for line in stat.lines() { + if line.starts_with(key) { + if let Some(val_str) = line.split_whitespace().nth(1) { + return val_str.parse().unwrap_or(0); + } + } + } + 0 +} + +/// Parse uptime seconds from /proc/uptime ("42.50 0.00\n"). +fn parse_uptime(data: &[u8]) -> (u64, u64) { + let s = core::str::from_utf8(data).unwrap_or(""); + let first = s.split_whitespace().next().unwrap_or("0.0"); + if let Some(dot) = first.find('.') { + let secs: u64 = first[..dot].parse().unwrap_or(0); + let frac: u64 = first[dot + 1..].parse().unwrap_or(0); + (secs, frac) + } else { + (first.parse().unwrap_or(0), 0) + } +} + +/// Completion code to human-readable name. +fn cc_name(cc: u64) -> &'static str { + match cc { + 0 => "INVALID", + 1 => "SUCCESS", + 2 => "DATA_BUF_ERR", + 3 => "BABBLE", + 4 => "USB_XACT_ERR", + 5 => "TRB_ERR", + 6 => "STALL", + 7 => "RESOURCE_ERR", + 8 => "BANDWIDTH_ERR", + 9 => "NO_SLOTS", + 10 => "INVALID_STREAM_TYPE", + 11 => "SLOT_NOT_ENABLED", + 12 => "EP_NOT_ENABLED", + 13 => "SHORT_PKT", + 14 => "RING_UNDERRUN", + 15 => "RING_OVERRUN", + 16 => "VF_EVENT_RING_FULL", + 17 => "PARAMETER_ERR", + 24 => "CMD_RING_STOPPED", + 25 => "CMD_ABORTED", + 26 => "STOPPED", + _ => "UNKNOWN", + } +} + +/// Parse the xHCI trace text and extract summary statistics. +struct XhciSummary { + total_records: u64, + cmd_complete: u64, + xfer_event: u64, + note: u64, + cmd_submit: u64, + xfer_submit: u64, + other: u64, + last_xfer_cc: u64, + last_xfer_slot: u64, + last_xfer_ep: u64, + // Diagnostic counters from XHCI_DIAG section + poll_count: u64, + event_count: u64, + consumed_xfer_enum: u64, + first_xfer_cc: u64, + first_queue_src: u64, + endpoint_resets: u64, +} + +fn parse_xhci_trace(data: &[u8]) -> XhciSummary { + let mut summary = XhciSummary { + total_records: 0, + cmd_complete: 0, + xfer_event: 0, + note: 0, + cmd_submit: 0, + xfer_submit: 0, + other: 0, + last_xfer_cc: 0, + last_xfer_slot: 0, + last_xfer_ep: 0, + poll_count: 0, + event_count: 0, + consumed_xfer_enum: 0, + first_xfer_cc: 0, + first_queue_src: 0, + endpoint_resets: 0, + }; + + let text = core::str::from_utf8(data).unwrap_or(""); + + // Extract total from header: "=== XHCI_TRACE_START total=N ===" + for line in text.lines() { + if line.starts_with("=== XHCI_TRACE_START total=") { + if let Some(rest) = line.strip_prefix("=== XHCI_TRACE_START total=") { + if let Some(num_str) = rest.split_whitespace().next() { + summary.total_records = num_str.parse().unwrap_or(0); + } + } + continue; + } + + // Parse trace record lines: "T NNNN OP_NAME S=SS E=EE TS=... LEN=..." + if !line.starts_with("T ") { + continue; + } + + let parts: Vec<&str> = line.split_whitespace().collect(); + if parts.len() < 6 { + continue; + } + + let op = parts[2]; // OP_NAME + match op { + "CMD_COMPLETE" => summary.cmd_complete += 1, + "XFER_EVENT" => summary.xfer_event += 1, + "NOTE" => summary.note += 1, + "CMD_SUBMIT" => summary.cmd_submit += 1, + "XFER_SUBMIT" => summary.xfer_submit += 1, + _ => summary.other += 1, + } + + // For XFER_EVENT and CMD_COMPLETE, extract slot and endpoint from S=SS E=EE + if op == "XFER_EVENT" || op == "CMD_COMPLETE" { + // parts[3] is "S=NN", parts[4] is "E=NN" + if let Some(s_val) = parts[3].strip_prefix("S=") { + summary.last_xfer_slot = s_val.parse().unwrap_or(0); + } + if let Some(e_val) = parts[4].strip_prefix("E=") { + summary.last_xfer_ep = e_val.parse().unwrap_or(0); + } + + // For XFER_EVENT, the completion code is in the TRB payload. + // The TRB data is on the NEXT line as hex. The CC is in bits 31:24 + // of the 3rd dword (bytes 8-11). We can look at subsequent hex line. + // However, parsing raw hex from the dump is complex. For a simpler approach, + // we check the payload bytes if available on the next line. + // For now, we'll leave last_xfer_cc=0 and populate it from hex if we can. + } + } + + // Parse XHCI_DIAG section for diagnostic counters. + let mut in_diag = false; + for line in text.lines() { + if line.starts_with("=== XHCI_DIAG ===") { + in_diag = true; + continue; + } + if line.starts_with("=== XHCI_DIAG_END ===") { + break; + } + if in_diag { + let parts: Vec<&str> = line.split_whitespace().collect(); + if parts.len() >= 2 { + let val: u64 = parts[1].parse().unwrap_or(0); + match parts[0] { + "poll_count" => summary.poll_count = val, + "event_count" => summary.event_count = val, + "consumed_xfer_enum" => summary.consumed_xfer_enum = val, + "first_xfer_cc" => summary.first_xfer_cc = val, + "first_queue_src" => summary.first_queue_src = val, + "endpoint_resets" => summary.endpoint_resets = val, + _ => {} + } + } + } + } + + // Second pass: extract completion codes from XFER_EVENT payload lines. + // Each XFER_EVENT record is followed by 1-2 hex dump lines (16 bytes = TRB). + // Completion code is in the "completion" dword: byte 8..12, bits 31:24. + // Format: " XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX" + let mut in_xfer_event = false; + for line in text.lines() { + if line.starts_with("T ") { + let parts: Vec<&str> = line.split_whitespace().collect(); + in_xfer_event = parts.len() >= 3 && parts[2] == "XFER_EVENT"; + continue; + } + + if in_xfer_event && line.starts_with(" ") { + // Parse the hex line. We need the 3rd dword (bytes 8-11). + // Format: " AABBCCDD EEFFGGHH IIJJKKLL MMNNOOPP" + let hex_parts: Vec<&str> = line.trim().split_whitespace().collect(); + if hex_parts.len() >= 3 { + // 3rd dword is hex_parts[2], CC is top byte (bits 31:24) + if let Some(dword_str) = hex_parts.get(2) { + if dword_str.len() >= 2 { + // First two hex chars are the CC (big-endian display of LE bytes) + // Actually the hex dump shows raw bytes in order. For a TRB: + // dword[2] at offset 8 displayed as bytes 08 09 0A 0B + // The completion code is the highest byte of the dword, + // which in little-endian is byte 11 (the 4th byte of dword 2). + // In the hex dump: bytes are in memory order. + // So for "IIJJKKLL", CC = LL (last byte pair). + let len = dword_str.len(); + if len >= 2 { + let cc_hex = &dword_str[len - 2..]; + if let Ok(cc) = u64::from_str_radix(cc_hex, 16) { + summary.last_xfer_cc = cc; + } + } + } + } + } + in_xfer_event = false; + } + } + + summary +} + +fn main() { + // Read procfs files + let uptime_data = read_procfs("/proc/uptime"); + let stat_data = read_procfs("/proc/stat"); + let xhci_data = read_procfs("/proc/xhci/trace"); + + // Parse uptime + let (up_secs, up_frac) = parse_uptime(&uptime_data); + + // Parse /proc/stat counters + let stat_str = core::str::from_utf8(&stat_data).unwrap_or(""); + let syscalls = parse_stat_value(stat_str, "syscalls"); + let irqs = parse_stat_value(stat_str, "interrupts"); + let ctx = parse_stat_value(stat_str, "context_switches"); + let forks = parse_stat_value(stat_str, "forks"); + let execs = parse_stat_value(stat_str, "execs"); + let cow = parse_stat_value(stat_str, "cow_faults"); + + // Parse xHCI trace + let xhci = parse_xhci_trace(&xhci_data); + + // Print compact summary + print!("=== btrace @ {}.{}s ===\n", up_secs, up_frac); + print!( + "sys={} irq={} ctx={} fork={} exec={} cow={}\n", + syscalls, irqs, ctx, forks, execs, cow, + ); + + if xhci.total_records > 0 { + print!( + "xhci: {} records, last_cc={} ({}) slot={} ep={}\n", + xhci.total_records, + xhci.last_xfer_cc, + cc_name(xhci.last_xfer_cc), + xhci.last_xfer_slot, + xhci.last_xfer_ep, + ); + print!( + "xhci: {} CMD_COMPLETE {} XFER_EVENT {} NOTE\n", + xhci.cmd_complete, xhci.xfer_event, xhci.note, + ); + print!( + "xhci: poll={} evt={} xe={} rst={} qsrc={} fcc={}\n", + xhci.poll_count, xhci.event_count, + xhci.consumed_xfer_enum, xhci.endpoint_resets, + xhci.first_queue_src, xhci.first_xfer_cc, + ); + // Dump raw XHCI_DIAG section for new diagnostic fields + let xhci_text = core::str::from_utf8(&xhci_data).unwrap_or(""); + let mut in_diag = false; + for line in xhci_text.lines() { + if line.starts_with("=== XHCI_DIAG ===") { + in_diag = true; + continue; + } + if line.starts_with("=== XHCI_DIAG_END ===") { + break; + } + if in_diag && !line.starts_with("poll_count") + && !line.starts_with("event_count") + && !line.starts_with("consumed_xfer") + && !line.starts_with("first_xfer_cc ") + && !line.starts_with("first_queue_src") + && !line.starts_with("endpoint_resets") + { + print!("xhci: {}\n", line); + } + } + } else { + print!("xhci: no trace records\n"); + } + + print!("=====================\n"); + std::process::exit(0); +} diff --git a/userspace/programs/src/init.rs b/userspace/programs/src/init.rs index 28c64bbf..9322e01f 100644 --- a/userspace/programs/src/init.rs +++ b/userspace/programs/src/init.rs @@ -107,7 +107,7 @@ fn main() { } print!("[init] TEST: all 5 iterations completed successfully\n"); - // Main loop: reap zombies and respawn crashed services. + // Main loop: reap zombies, respawn crashed services. let mut status: i32 = 0; loop { match waitpid(-1, &mut status as *mut i32, WNOHANG) {