Skip to content

feat(usb)!: replace the USB device stack with the endpoint core - #370

Draft
Jiu-xiao wants to merge 28 commits into
devfrom
refactor/usb-core
Draft

Jiu-xiao wants to merge 28 commits into
devfrom
refactor/usb-core

Conversation

@Jiu-xiao

@Jiu-xiao Jiu-xiao commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Replaces the USB device stack with a new endpoint core. Endpoints own their
memory (slots), completion callbacks run once each, in submission order, inside a
single owner, and stopping an endpoint never waits for the bus. The device core,
six class families and the backends of five platforms move onto it.

Example: a CDC class used to keep its own buffer and wait for the endpoint to be
idle before sending. Now it borrows a slot from the endpoint, fills it and hands
it back; the hardware sends it when it can and the class gets one callback.

Draft: the boards have not been rerun on the final head yet, and the code outside
the library (board packages, code generator, product firmware) is not ported.

Commits (28, each builds and passes the host tests)

# Commits What to read
1 remove Deletion only: old device core, classes, three backends and the old CDC tests
2 endpoint core core/ep.hpp holds the whole contract: application API, backend contract (StartHardware / StopHardware / ClearStallHardware / OnHardwareComplete), HwLimits
3 device core dev_core.hpp (what a backend calls and implements), device_class.hpp (class interface and endpoint helpers)
4-9 CDC, HID, UAC1, DFU, gs_usb, CMSIS-DAP One commit per class with its tests. DAP keeps its command engine; only transport changes
10 FSDEV Register runtime shared by STM32 FS/DRD and CH32 USBD
11-14 STM32, CH32, ESP32, HPM Platform backends; each header starts with the hardware defects, the workaround and the measurement behind it
15 FSDEV isochronous "Isochronous endpoints" in the FSDevEndpoint class notes
16 D+ low on detach; CH32 USBD EP0 gate Detach turns the clock off and drives PA12 low (boards with a fixed D+ pull-up); EnableEp0BulkGate
17 COUNT_RX latch; closed endpoints NAK EnableRxCountLatch / EnableRxCountSweep; CloseHardware writes NAK (isochronous: DISABLED)
18 Non-blocking detach core/usb_detach_timer.hpp; tick sources in the system layer (system/*/tick.cpp, stm32_hal_tick.cpp)
19 USB simulation Tests only: simulated controller under the real cores and classes, USB/IP end-to-end scripts
20 Comments Four comments had the Chinese and English paragraphs on one line
21 DAP padding DAP v2 Execute/QueueCommands ignore bytes after the last command, as the CMSIS-DAP reference does
22 Contract notes StartHardware after an unreported bus reset; disconnect is reported last
23 Endpoint::Ahead No default member values: EndRequests initialised a 7-entry local array on every call (16 us per call on a 72 MHz F103)
24 32-bit atomics USB flags and the CH32 USB/CAN shared flags become std::atomic<uint32_t>; Endpoint::State is uint32_t; ch32_atomic_shim.c is back to __atomic_compare_exchange_4 only (commit 12 had added a one-byte exchange)
25 SettleStop; tick comments CH32 OTG FS/HS: Outcome(bool, ...) becomes SettleStop(const StopEvent&, bool), no behaviour change; comments say the tick pends the controller interrupt instead of "wakes the owner"
26 No copy at a stop CH32 OTG FS: a packet collected at a stop is counted but no longer copied into the slot (a stopped OUT request is delivered ABORTED without a slot)
27 gs_usb frames No per-frame zeroing of the 76-byte local frame and the CAN packs; data is zeroed past its length only, since the full 8 or 64 data bytes go on the wire
28 STM32 FIFO words FillFifo copies whole words with a constant length (one unaligned load) instead of a memcpy call per FIFO word; FIFO mode only

Application changes

  • Backend class names are unchanged (STM32USBDeviceOtgFS, ...); constructor
    arguments changed, see each class.
  • Bare-metal CH32 and HPM projects call RegisterTickSource() once at start-up
    and OnTick() from the system tick interrupt; constructing a USB device without
    a tick source fails a REQUIRE. FreeRTOS, ThreadX, ESP-IDF and bare-metal STM32
    need no change.
  • ESP-IDF projects need CONFIG_FREERTOS_USE_TIMERS (checked at compile time).
  • CH32 USBD (CH32V20x full-speed device) needs SYSCLK of 96 MHz or more, see
    below.

Verification

Host (Linux):

  • The four CI configurations (Debug/Release x DEV_ASSERT x Make/Ninja): 0
    warnings, ctest 16/16 (test_linux_can skipped without vcan).
  • tools/format_cpp_files.sh --check, tools/format_cmake_files.sh --check: pass.
  • Doxygen 1.12.0: 531 warnings, baseline 933.
  • Mutation checks (each mechanism removed in turn) are caught by the tests.
  • USB/IP end-to-end against the Linux kernel drivers: CDC, HID, gs_usb, DFU, UAC;
    plug cycles 20/20; OpenOCD, pyOCD and probe-rs against the simulated DAP.

Boards (before the last commit; every board restored to its original firmware and
read back):

Board Backend Forced reset in the owner Stop / start Other
APM32F103 FSDEV 50/50 20/20 CDC 2 MiB; isochronous IN 180 s, OUT 60 s clean
STM32H503 FSDEV (DRD) 50/50 20/20 isochronous 60 s clean
CH32V203 FSDEV (CH32 USBD) - - 96 / 144 MHz: control writes 300/300, CDC no loss
CH32V307 CH32 USBHS 50/50 20/20 detach 5.00-5.99 ms; CDC 2 MiB
CH32V208 CH32 USBFS 50/50 20/20 detach 5.03-5.92 ms; CDC 2 MiB
HPM5301EVKLite HPM 50/50 20/20 detach 10.02-10.99 ms; CDC 2 MiB
STM32H723, FIFO and DMA STM32 OTG 20/20 20/20 10 MiB loopback; 60 s control load 0 errors
ESP32-S3 ESP32 20/20 20/20 10 MiB loopback; isochronous IN 60 s clean; 317k control transfers 0 errors

The last commit was measured on the APM32F103: the idle path of EndRequests
goes from 19.8 us to 4.9 us, and an idle control load does 50.8k transfers in
20 s instead of 48.2k.

Known limits

  • CH32 USBD at 48 MHz: a SETUP not read within about 15 us can be overwritten
    by a data packet (about 12 % of control writes fail), and a NAKed OUT packet
    can leave COUNT_RX at 0x3FE (bulk OUT loses packets). Neither happens at 96 or
    144 MHz; the class notes require 96 MHz or more.
  • CH32V307 USBHS, isochronous IN next to bulk stress: in 3 of 60 one-minute runs
    the device skipped one frame (one host transaction error, no samples lost).
    Cause not located.
  • APM32F103 idle control load is still about 3.6 % below the old stack.
  • CH32 USBHS packet limits: interrupt 64, isochronous 1023 bytes.
  • Not tested on hardware: CH32V208 suspend/resume and isochronous OUT on a
    Windows host.

Not in this PR

The four board packages, the code generator, the website documentation and the
XR-DAP product firmware still use the old API and are updated after this branch
is settled.

…f their replacement

The following commits replace the USB device stack bottom-up. This one removes
what cannot be kept while that happens: the device core, the class base and the
composition, every class except the DAP headers (which are changed in place
later), the STM32, CH32 and ESP32 backends and the CDC test. The endpoint and
descriptor files under core/ stay until the next commit replaces them, so the
library still builds.

The compile check with assertions disabled includes a CDC header; the include
is dropped here and returns with the CDC classes.
…ait for the bus

Endpoint now owns its transfer memory as 1 to 8 slots. A caller borrows a slot,
fills it and queues it; requests complete exactly once, in submission order,
with the callback run by the owner. The owner is a serialized service
(DeviceService) entered from the controller interrupt or from a thread, so the
endpoint needs no lock and never masks interrupts.

Contract towards a backend (ep.hpp): StartHardware / StopHardware /
ClearStallHardware and one OnHardwareComplete per request. A stop returns at
once; a backend that has to wait for the bus answers PENDING and reports later.
HwLimits tells the core how many requests the hardware takes at a time and how
many packets fit a service interval; the core keeps the rest queued.

EndpointSlots holds the slot states in one atomic word, with a per-slot stamp
that invalidates stale handles.

Tests: test_usb_ep_slots, test_usb_ep, test_usb_ep_pool.
DeviceCore runs enumeration and control transfers inside the owner; EP0 is two
ordinary endpoints. A backend reports bus events (reset, suspend, resume, SETUP,
start of frame) through On... calls and implements the controller hooks
(StartController, StopController, ResetController, SetAddress).

DeviceClass is the interface of a class: descriptor shape and bytes, bind and
unbind of its endpoints, control requests with a real data stage, service
requests from any context. It carries the shared endpoint helpers
(WriteEndpointDescriptor, ConfigureEndpoint, BindEndpoint, UnbindEndpoint).
DeviceComposition lays out interfaces, endpoints, strings and BOS capabilities
of the configurations at construction; nothing is allocated afterwards.

Tests: test_usb_dev_core.
CDCBase, CDCUart, CDCToUart and the two throughput test classes. Receive slots
are handed to the reader in arrival order and held while the read port is full;
SetConfig from the device side goes to the owner through a single-writer
sequence lock.

Tests: test_cdc.
HID with keyboard, mouse and gamepad. Reports are sent from any context through
the IN endpoint's slots; GET_REPORT / SET_REPORT use the control data stage.

Tests: test_hid.
One isochronous request per service interval, paced by the start of frame. At
high speed the interval is configurable and an interval that finds every slot
taken is dropped and counted.

Tests: test_uac.
The state machine advances on the completion of the control status stage, so a
detach or a manifest starts only after the host has taken the answer. Flash work
runs outside the owner and is handed back through a job slot.

Tests: test_dfu.
Host frames are taken per slot and echoed in order; a full echo queue holds the
OUT slot so the host is NAKed instead of frames being dropped.

Tests: test_gsusb.
The command engine is unchanged. The transport borrows and queues slots instead
of owning endpoint buffers: v2 keeps every free OUT slot armed and answers from
the Step thread; v1 does the same over the HID interrupt endpoints.

Tests: test_dap.
…roller

FSDevDevice and FSDevEndpoint drive the EPnR / BTABLE / PMA controller of the
STM32 families with a USB FS device or DRD peripheral, of CH32 (USBD) and of
compatible parts. A platform supplies the register addresses, the PMA shape and
the interrupt lines. A stop parks the endpoint address while STAT is rewritten,
so a transaction in flight is neither lost nor counted twice; a halt cleared on
an armed endpoint resets the toggle and leaves the request armed. Isochronous
endpoints are not supported.

Tests: test_usb_fsdev.
STM32USBDeviceOtgFS / OtgHS drive the OTG core directly (FIFO or internal DMA);
STM32USBDeviceDevFs is the platform part of the FSDEV runtime. The HAL keeps
clocks, pins and the interrupt vectors: HAL_PCD_IRQHandler is wrapped at link
time and the device object refuses to start when the wrap is missing.

Constructor parameters changed, see the class comments.
CH32USBOtgFS, CH32USBOtgHS and CH32USBDeviceFS (USBD, on the FSDEV runtime). The
data toggle of bulk and interrupt endpoints is kept in software, which is what
makes a stop exact on these controllers; the comments in ch32_usb_endpoint.hpp
list the hardware behaviour this works around and the measurements behind it.

Tests: test_ch32_usb (the drivers on fake register blocks).

Constructor parameters changed, see the class comments.
ESP32USBDevice drives the DWC2 core of the ESP32-S3 (full speed) with the
internal DMA. The interrupt is allocated on the constructing core and masked by
rerouting its source, never by disabling interrupts.

Constructor parameters changed, see the class comments.
HPMUSBDevice drives the USB device controller of HPMicro parts (run on an
HPM5301), at high or full speed. Each slot has its own transfer descriptor, so
several requests of one endpoint can be on the hardware at a time.
The controller always double-buffers an isochronous endpoint (RM0008 23.4.4,
CH32 RM 21.2.3.2): the transmit and the receive descriptor of the number are the
two buffers, DTOG selects the one the hardware uses next and is toggled by the
hardware after every transaction, and STAT is only disabled or valid.

- Buffer 0 is the endpoint's own PMA, buffer 1 the PMA of the other direction of
  the number (both EPConfig entries, equal sizes); that direction cannot be
  configured while the isochronous endpoint is.
- Two requests on the hardware, one per buffer; one packet per request.
- A buffer without a request is sent as a zero-length packet, never stale data.
- Stop, Close and reset park the endpoint address, settle the requests from
  DTOG and CTR and return at once; a token arriving in that moment gets no
  answer.
- Stall writes STAT disabled; ClearStall from there empties both buffers and
  writes valid. ClearStall on a valid endpoint changes nothing, so armed
  requests keep their order.
- The wait for the status packet of a control read ends on an isochronous
  completion instead of latching it: the double buffer tolerates one frame.

DRD parts get the buffer-count accessors for their 32-bit descriptors.

Tests: test_usb_fsdev (fsdev_iso_* cases, the classic and the DRD layout).
Some boards carry a fixed pull-up on D+ (F103 boards of the Bluepill kind, the
STM32H503 test board), so switching the internal pull-up off does not detach and
the host keeps the old address after a device reset. Every detach (construction,
Stop, ResetController) now powers the transceiver down, switches the USB clock
off and drives PA12 low for DETACH_US before it attaches again.

- PA12 is D+ on every supported STM32 family and package (pin data of 887 parts);
  LIBXR_STM32_FSDEV_DP_DETACH / LIBXR_CH32_USBD_DP_DETACH 0 switch the drive off.
- Measured on F103 and H503: the pin reads 0 while driven, the host logs a
  disconnect and enumerates again; a control image without the drive is not seen.
- CH32V203 USBD takes and ACKs every OUT data packet while EP0 is typed CONTROL.
  EnableEp0BulkGate types EP0 BULK for the data stage of a control write and
  reads the SETUP first thing at the interrupt entry.
- CH32 USBD raises the isochronous completion on the high-priority line; both
  lines enter Dispatch at the same priority.

Tests: test_usb_fsdev (detach and EP0 gate cases).
- CH32V203 USBD writes COUNT_RX = 0x3FE when, while an OUT packet waits to be
  read, an IN is ACKed and a later OUT packet is NAKed. EnableRxCountLatch (on by
  default on CH32 USBD) latches the count when the endpoint is handled, before
  its IN completion; EnableRxCountSweep latches every pending OUT endpoint. Both
  this window and EP0's SETUP window shrink with the interrupt handling: at 48 MHz
  bulk OUT still lost packets, at 96 and 144 MHz none, so CH32 USBD needs SYSCLK
  of 96 MHz or more (class notes).
- Close writes NAK instead of DISABLED on every FSDEV part: after STAT_TX went
  through DISABLED, CH32 USBD occasionally sent a later IN packet corrupted. The
  DWC2 backends keep a closed endpoint answering NAK as well. An isochronous
  endpoint has no NAK and still closes DISABLED; configuring one sets the other
  direction of its number DISABLED first.

Tests: test_usb_fsdev (COUNT_RX latch and closed endpoint cases).
ResetController, Stop and construction no longer wait DETACH_MS with D+ held:
they hold the detach and return (F103 25.9 us, H503 5.75 us). USB::DetachTimer
counts the hold on the system tick and attaches again through the owner, on a
DriverAsyncCondition, never in the tick interrupt itself.

- The tick source belongs to the system layer: FreeRTOS wraps
  xTaskIncrementTick, ThreadX wraps _tx_timer_interrupt, ESP-IDF registers a
  FreeRTOS tick hook on CPU 0 and attaches through the timer service task,
  bare-metal STM32 wraps HAL_IncTick (system none only), and bare-metal CH32 and
  HPM boards call RegisterTickSource and OnTick. A second registration is fatal;
  a missing tick source is checked when a detach starts.
- FSDEV masks its own interrupt line while D+ is held with the clock off (the
  request stays active with the clock off, measured on F103) and the tick unmasks
  and pends it; DeviceService::MaskInterrupt states this exception.
- Connect on F1/F3 reuses the CubeMX PCD handle (STM32USBInitHandle).
- Every backend that waited (FSDEV, DWC2 on STM32 and ESP32-S3, CH32 OTG FS and
  HS, HPM) now holds the detach on the same DetachTimer.

Tests: test_usb_detach_timer, test_usb_fsdev and test_usb_ep (detach cases),
test_ch32_usb.
The real endpoint core, device core and device classes run on a simulated
controller. The judgement of every case comes from the contracts written in
ep.hpp, usb_service.hpp and dev_core.hpp.

- ctest: usb_sim_contract (23 cases), usb_sim_profiles (FSDEV, OTG FS, ESP32-S3
  and HPM high-speed limits), swd_target, dap_fuzz, dap_replay (sessions recorded
  from OpenOCD, pyOCD and probe-rs).
- By hand (root and vhci-hcd): usb_sim_usbip, usb_sim_classes and dap_sim_usbip
  export the simulated device over USB/IP to the Linux kernel; classes_e2e.sh
  runs CDC, HID, gs_usb, DFU and UAC through the kernel drivers, plug_cycle.sh
  attaches and detaches 20 times, dap_e2e.sh runs OpenOCD, pyOCD and probe-rs
  with and without injected WAITs.
- USB_SIM_ABLATION=n breaks the simulation in one place; each of the 21 places
  is caught by its case.

Library sources are unchanged; test/automatic/CMakeLists.txt includes
driver/usb/sim/usb_sim.cmake.
Four comments ran the end of the Chinese paragraph and the start of the English
one into the same line. No code changes.
DAP_ExecuteCommands and DAP_QueueCommands answered DAP_ERROR when the request
carried bytes after its last command, as a host that pads every request to the
packet size sends. The CMSIS-DAP reference ignores those bytes; both checks
that wanted the commands to fill the request exactly now accept trailing bytes.
A count larger than the commands present is still an error.

Tests: test_dap (padded QUEUE / EXECUTE), dap_fuzz now pads these two commands
as well.
- StartHardware: it may be called after the hardware has latched a bus reset or
  disconnect that is not reported yet (starts come at the end of an owner pass).
  A backend may refuse (STM32 OTG and HPM for endpoints other than EP0) or
  accept and let the Close of the reset report stop it, at once or later
  (FSDEV at the next SOF). ESP32-S3 accepts: measured 50 times, the chip keeps
  the enable bit on the deactivated endpoint but sends nothing, and the stop
  after the reset report ends without a timeout.
- DeviceCore: disconnect is reported last; when an unplug, a replug and the
  host's reset fall into one status read, the device stays deinitialized until
  the host's next reset. STM32 OTG is the only backend reporting a disconnect.

No code changes.
EndRequests builds a local array of Ahead with MAX_SLOTS - 1 entries on every
call and uses only the first ahead_count_ of them. With default member values
the whole array was initialised each time; with newlib-nano that is a byte-wise
memcpy of 84 bytes. DeviceCore::OnSetup flushes both EP0 directions, so every
SETUP paid for it twice.

On an F103 at 72 MHz the idle path of EndRequests (nothing on the hardware)
goes from 19.8 us to 4.9 us, and an idle control load does 50.8k transfers in
20 s instead of 48.2k.

The members are now written in full where an Ahead is created; the one place
that relied on the defaults names the empty value. No behaviour change.
LibXR keeps its atomic variables 32 bits wide. The USB flags were
std::atomic<bool>, <uint8_t> and <uint16_t>; on CH32 (QingKe V4 has word AMOs
only) the bool exchanges in dev_core, device_composition and cdc_uart became
calls to __atomic_exchange_1, which the CH32 backend commit had added to
ch32_atomic_shim.c.

The flags are now std::atomic<uint32_t> holding 0 or 1, Endpoint::State has
uint32_t as its underlying type, and every memory order is unchanged. The
USB/CAN shared state flags of CH32 (ch32_usbcan_shared.hpp) move to
std::atomic<uint32_t> as well.
ch32_atomic_shim.c is back to the dev version (__atomic_compare_exchange_4 only).
CH32 OTG FS/HS endpoints: Outcome(bool, ..., bool) becomes
SettleStop(const StopEvent&, bool). The function counts the last completion
event of a stopped request (toggle, copy into the slot, bytes) and gives its
result; the event's fields are named at the call sites, and {} means no event.
No behaviour change.

Comments and test notes no longer say the tick "wakes the owner": the tick
publishes an event and pends the controller interrupt, which runs another pass
of ServiceHardware (usb_service.hpp, ep.hpp, the CH32 HS backend, HPM).
CH32 OTG FS: SettleStop no longer copies the packet collected at a stop from
the shared block into the slot, and loses its may_copy parameter. The core
delivers a stopped OUT request as ABORTED and lends no slot, so the copied
bytes were never read. The bytes are still counted; what the class sees
(result, actual, no slot) is unchanged.
Each CAN frame zeroed a 76-byte QueueItem (receive, FD receive, pump and
echo) and the CAN pack of a host frame before overwriting it. The locals are
now left uninitialised; where data is filled, the bytes past its length are
set to 0, because Pack sends the full 8 (classic) or 64 (FD) data bytes.

Tests: a classic and an FD frame after a frame of 0xFF bytes are 0 past their
length on the wire.
FillFifo copied each word with memcpy(&value, src, min(4, left)); the length
is not a constant, so the copy stayed a call to memcpy for every FIFO word
(16 per full-speed packet, 128 per high-speed packet; Arm GNU 14.2, -O2 and
-Os, Cortex-M4/M7). Whole words now use a constant length, which compiles to
one unaligned load, and the part short of a word is copied once. ReadFifo
already worked this way. Only FIFO mode (no DMA) uses this path.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant