Skip to content

perf(mem): add compile-time memory policies, 32-bit shifted copy, and FastSet -O3 fix - #371

Draft
Jiu-xiao wants to merge 4 commits into
devfrom
perf/fastcopy-shifted
Draft

Jiu-xiao wants to merge 4 commits into
devfrom
perf/fastcopy-shifted

Conversation

@Jiu-xiao

@Jiu-xiao Jiu-xiao commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Summary

Four commits on top of dev (c84150262) introducing independent compile-time dispatch policies for Memory::{FastCopy,FastMove,FastSet,FastCmp}, 32-bit shift-and-combine FastCopy, a GCC -O3 peel/tail fix for FastSet, and a load-before-store burst fix in CopyAlignedWords:

  1. 333673853 — perf(mem): add compile-time FastCopy policies and 32-bit shifted copy
    • Introduce compile-time LIBXR_FASTCOPY_POLICY dispatch (SHIFTED_ODD, SHIFTED_ALL, STD_LIBC, UNALIGNED_WORD), CopyAlignedWords<WordT>, CopyShiftedLittleEndian<1U..3U>, and HAS_FAST_UNALIGNED_WORD-guarded UNALIGNED_WORD.
  2. fd6481d44 — test(mem): add FastCopy policy matrix and boundary regressions
    • Add (nullptr, nullptr, 0), non-overlapping FastMove, and 12×18 phase/boundary sweep coverage in test_mem.cpp, plus standalone policy matrix tests in test_mem_policy.cpp.
  3. ff150d818 — perf(mem): add per-op memory policies and fix FastSet -O3 peel
    • Extract policy definitions into src/core/libxr_mem_policy.hpp (wrapped in /// \cond../// \endcond) and add independent per-operation policies + LIBXR_MEM_WORD_SIZE (4 or 8 bytes):
      • LIBXR_FASTCOPY_POLICY: SHIFTED_ODD (0, core default) / SHIFTED_ALL (1, driver/ch default) / STD_LIBC (2) / UNALIGNED_WORD (3, opt-in, protected by static_assert on __ARM_FEATURE_UNALIGNED / x86 / arm64).
      • LIBXR_FASTMOVE_POLICY, LIBXR_FASTSET_POLICY, LIBXR_FASTCMP_POLICY: BUILTIN (0, core default) / STD_LIBC (1).
    • Configure independent resolution/validation in cmake/config.cmake (FATAL_ERROR on invalid values; prints Memory policies: copy=..., move=..., set=..., cmp=...) and support ACTIVE_FASTCMP_POLICY in src/core/libxr_mem_cmp.cpp (keeping (size == 0U) || (a == b) guard under STD_LIBC).
    • Platform/driver defaults:
      • driver/ch: Copy=SHIFTED_ALL, Move/Set/Cmp=BUILTIN.
      • driver/esp Xtensa (CONFIG_IDF_TARGET_ARCH_XTENSA or IDF_TARGET in esp32/esp32s2/esp32s3): Copy=STD_LIBC, Set=STD_LIBC, Move=BUILTIN, Cmp=BUILTIN (ESP RISC-V inherits Copy=SHIFTED_ODD, Move/Set/Cmp=BUILTIN).
      • driver/{linux,webots,webasm}: all 4 default to STD_LIBC.
      • driver/{st,mspm0,hpm} & none: Copy=SHIFTED_ODD, Move/Set/Cmp=BUILTIN.
    • Fix GCC -O3 (-ftree-loop-distribute-patterns) FastSet regression in src/core/libxr_mem_o3.cpp by replacing tail/head < WORD_SIZE byte loops with loop-free FillSmallBytes<WORD_BYTES>, eliminating emitted external memset calls.
    • Document all 4 options in README.md and expand test/automatic/CMakeLists.txt to 7 -O3 -Wall -Wextra -Werror policy matrix targets (including test_mem_policy_mixed_esp_xtensa).
  4. 85d299af7 — perf(mem): batch 8-word loads before stores in CopyAlignedWords
    • Load all 8 words (a0..a7) before writing back (dw[0..7]) in CopyAlignedWords<WordT>.
    • Eliminates GCC -O3 runtime alias-check fallback loops on Cortex-M3/M4/M7 (alias_cmp: 3 -> 0), restores ldrd/strd bursts for short forward-overlap FastMove (s - d < 32 B), preserves 4x unaligned ldr.w/str.w merge for uint16_t on ARMv7-M, and trims ARMv7-M Flash footprint by ~332–336 B (~544 B on x86_64).

Behavior Changes

  • LibXR::Memory::{FastCopy,FastMove,FastSet,FastCmp} dispatch independently at compile time (if constexpr, zero runtime overhead) according to LIBXR_FAST*_POLICY.
  • Hosted targets (linux/webots/webasm) delegate all 4 functions to libc by default; ESP Xtensa targets delegate FastCopy and FastSet while keeping FastMove and FastCmp built-in.
  • Zero-length (including nullptr), self-operation (dst == src / a == b), and overlap semantics remain unchanged.

Verification (on 85d299af7)

  • Formatting, patch & Doxygen:
    • clang-format 21.1.8 & git diff --check: clean (cmake-format 0.6.13 verified on untouched CMake files since ff150d818).
    • doxygen 1.12.0: 933 warnings (matches doc/doxygen_warning_baseline.txt baseline 933; 0 in touched files).
  • CI build & test matrix (Unix Makefiles Debug/ON, Release/ON, Release/OFF, Ninja Debug/ON, -DLIBXR_TEST_BUILD=ON):
    • 0 compiler warnings across all 4 configs; all 4 pass 16/16 active tests (17 total; test_linux_can skipped without vcan).
    • LIBXR_SYSTEM=none Release builds with 0 warnings (Memory policies: copy=SHIFTED_ODD, move=BUILTIN, set=BUILTIN, cmp=BUILTIN).
  • ASan + UBSan: 7/7 test_mem_policy_* targets (shifted_odd_32, shifted_all_32, shifted_odd_64, shifted_all_64, std_libc, mixed_esp_xtensa, unaligned_word) pass with 0 findings.
  • Cross-compile .text + .literal Flash footprint vs dev (c84150262) (-O3/-O2 -ffunction-sections -Wall -Wextra -Werror, .rodata = 0 -> 0 B, 0 warnings):
    • Verified 0 external memset calls inside FastSet across all 7 targets (m0p, m3, m4, m7, rv32, s3, x86), and alias_cmp = 0 on m3/m4/m7 with ldrd/strd and unaligned ldr.w/str.w merge preserved:
      • cortex-m0plus (SHIFTED_ODD + BUILTIN): 2188 -> 2680 B (+492 B: Copy 764->1248, Move 896->884, Set 240->260, Cmp 288->288).
      • cortex-m3 (SHIFTED_ODD + BUILTIN): 2140 -> 2432 B (+292 B: Copy 872->1144, Move 748->748, Set 224->244, Cmp 296->296).
      • cortex-m4 (SHIFTED_ODD + BUILTIN): 2152 -> 2440 B (+288 B: Copy 876->1144, Move 752->752, Set 224->244, Cmp 300->300).
      • cortex-m7 (SHIFTED_ODD + BUILTIN): 2136 -> 2424 B (+288 B: Copy 876->1148, Move 744->740, Set 224->244, Cmp 292->292).
      • rv32imac (SHIFTED_ALL + BUILTIN): 2128 -> 2512 B (+384 B: Copy 772->1178, Move 776->772, Set 242->222, Cmp 338->340).
      • esp32s3 (Xtensa Copy/Set=STD_LIBC, Move/Cmp=BUILTIN): 2091 -> 1223 B (-868 B: Copy 698->39, Move 828->821, Set 226->24, Cmp 339->339).
      • x86_64 (Linux all STD_LIBC): 4561 -> 102 B (-4459 B: 25/25/25/27 B).
      • Opt-in UNALIGNED_WORD + BUILTIN on m3/m4/m7: FastCopy 160 B, totals 2140->1616 B (-524 B), 2152->1624 B (-528 B), 2136->1604 B (-532 B), alias_cmp = 0.
    • Verified on ff150d818 prior to 85d299af7 (unchanged policy logic): UNALIGNED_WORD static_assert positive (m3, m7, x86) and negative (m0p, m3 -mno-unaligned-access, rv32, s3), plus Linux -DLIBXR_FASTCMP_POLICY=BUILTIN and -DLIBXR_FASTSET_POLICY=BOGUS FATAL_ERROR.

Silicon Benchmarks (prior commits fd6481d44 / ff150d818)

  • 5-board + x86_64 baseline (FastCopy/FastMove/FastSet/FastCmp vs libc, 596/596 sweep checks pass):
    • ESP32-S3 (XR-DAP-S3 @ 240 MHz): ROM memcpy beats FastCopy (0.59x–1.08x), ROM memset ties/beats FastSet (0.94x–1.10x aligned, 0.43x–0.86x unaligned + saves 202–211 B Flash), whereas ROM memmove is byte-by-byte (FastMove 8.86x–19.12x faster) and ROM memcmp loses at >=64 B (FastCmp 1.74x–7.91x faster on eq_0_0/eq_1_1), justifying driver/esp Xtensa Copy=STD_LIBC, Set=STD_LIBC, Move=BUILTIN, Cmp=BUILTIN.
    • x86_64 Linux: glibc AVX2 outperforms built-ins across all 4 primitives, justifying driver/{linux,webots,webasm} all STD_LIBC.
    • FastCopy (SHIFTED_ODD/SHIFTED_ALL/UNALIGNED_WORD): Cortex-M0+ (STM32G0B1, 1024 B odd) 3.27x–3.43x; RV32IMAC (CH32V203, 1024 B misaligned) 1.27x–3.41x; Cortex-M7 (STM32H723, 1024 B) 1.34x–2.03x (DTCM) / 1.34x–1.91x (AXI); Cortex-M3 (APM32F103CB) keeps (0,2) non-regressing (1565 vs 1552) under SHIFTED_ODD, with opt-in UNALIGNED_WORD 12–64 B in 49–163 cycles (1.30x–3.06x vs SHIFTED_ODD).
  • FastSet FillSmallBytes fix re-benchmark (old -> new / libc cycles & new ratio vs memset, 596/596 sweep pass):
Board Offset 12 B 16 B 64 B 256 B 1024 B
APM32F103CB (M3 72 MHz) off_0 78->73 / 152 (2.082x) 84->82 / 200 (2.439x) 100->102 / 776 (7.608x) 220->234 / 3080 (13.162x) 700->762 / 12296 (16.136x)
APM32F103CB off_1 129->106 / 152 (1.434x, was 0.907x) 135->109 / 200 (1.835x) 185->146 / 776 (5.315x) 305->278 / 3080 (11.079x) 785->806 / 12296 (15.256x)
STM32G0B1 (M0+ 64 MHz) off_0 86->79 / 118 (1.494x) 91->84 / 150 (1.786x) 110->114 / 534 (4.684x) 236->240 / 2070 (8.625x) 740->744 / 8214 (11.040x)
STM32G0B1 off_1 142->113 / 118 (1.044x, was 0.796x) 147->118 / 150 (1.271x) 199->167 / 534 (3.198x) 320->290 / 2070 (7.138x) 800->770 / 8214 (10.668x)
ESP32-S3 (LX7 240 MHz) off_0 43->46 / 42 (0.913x) 46->49 / 43 (0.878x) 58->61 / 59 (0.967x) 112->115 / 119 (1.035x) 328->331 / 359 (1.085x)
ESP32-S3 off_1 126->54 / 55 (1.019x) 129->57 / 55 (0.965x) 156->84 / 68 (0.810x) 211->139 / 128 (0.921x) 427->355 / 368 (1.037x)

All tested boards were backed up via double-read and byte-identically restored to their intake flash state.

Add compile-time LIBXR_FASTCOPY_POLICY selection for Memory::FastCopy
while keeping zero-length (including null pointers), self-copy, and
64-bit auto-vectorized paths intact:

- SHIFTED_ODD (default): keeps 16-bit CopyAlignedWords<uint16_t> on even
  address differences and uses phase-specialized CopyShiftedLittleEndian<1/3>
  on 32-bit little-endian odd differences.
- SHIFTED_ALL (driver/ch default): dispatches all 32-bit little-endian
  misaligned phases 1..3 through CopyShiftedLittleEndian<1..3>.
- STD_MEMCPY (linux/webots/webasm and ESP Xtensa default): delegates to
  std::memcpy after zero-length and self-copy guards.
- UNALIGNED_WORD (opt-in): direct 4x unaligned 32-bit word copy guarded by
  HAS_FAST_UNALIGNED_WORD static_assert.
Add phase/boundary regressions to test_mem.cpp and register 6 standalone
test_mem_policy_* CTest targets covering SHIFTED_ODD (32/64-bit),
SHIFTED_ALL (32/64-bit), STD_MEMCPY, and UNALIGNED_WORD across zero-length
null pointers, self-copy, all 8x8 alignment pairs, sentinel guards, and
overlapping forward/backward FastMove.
- Split LIBXR_FASTCOPY_POLICY into libxr_mem_policy.hpp and add independent LIBXR_FASTMOVE_POLICY, LIBXR_FASTSET_POLICY, and LIBXR_FASTCMP_POLICY knobs (BUILTIN/STD_LIBC) alongside LIBXR_MEM_WORD_SIZE
- Default driver/{linux,webots,webasm} to STD_LIBC for all four operations; default driver/esp Xtensa to Copy/Set=STD_LIBC while keeping Move/Cmp=BUILTIN per ESP32-S3 ROM vs builtin benchmarks
- Add loop-free FillSmallBytes<WORD_BYTES> for FastSet head/tail to prevent GCC -O3 -ftree-loop-distribute-patterns from emitting external memset calls
- Expand policy tests to 7 CTest matrix targets including mixed_esp_xtensa and update CMake options documentation in README.md
@Jiu-xiao Jiu-xiao changed the title perf(mem): add compile-time FastCopy policies and 32-bit shifted copy perf(mem): add compile-time memory policies, 32-bit shifted copy, and FastSet -O3 fix Nov 20, 2026
- Load all 8 words before writing back in CopyAlignedWords<WordT>'s burst loop
- Eliminates GCC -O3 runtime alias-check fallback loops on Cortex-M3/M4/M7
- Restores ldrd/strd bursts for short forward-overlap FastMove (s - d < 32 B) and shrinks FastCopy+FastMove Flash footprint by ~332-336 B on ARMv7-M (and ~544 B on x86_64)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant