Skip to content

fix: flush CPU cache lines in device UUID header read/write paths#67

Merged
seohui-XCENA merged 3 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/device-header-cache-flush
Jul 23, 2026
Merged

fix: flush CPU cache lines in device UUID header read/write paths#67
seohui-XCENA merged 3 commits into
xcena-dev:mainfrom
seohui-XCENA:fix/device-header-cache-flush

Conversation

@seohui-XCENA

@seohui-XCENA seohui-XCENA commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

  • The device UUID header is read/written with plain CPU load/store through a DEV_DAX mmap. CPU stores stay in the write-back cache, and msync() does not flush CPU caches on DEV_DAX (no page cache, no fsync op).
  • CXL 2.x shared memory has no cross-host cache coherence, so on multi-host setups a freshly written header is invisible to other hosts — both RMs auto-initialize the same device with different UUIDs (identity split-brain) — and a header cached by an earlier read goes stale when another host rewrites it.
  • Multi-host simulation (HDM window aliasing) measured the CPU-store path stale 0/50 on every platform, corrected only by clflush+fence.

🏗️ Design Changes

  • Behavioral: header writes now end with clflush+mfence (write-back to device), header reads now start with clflush+mfence (invalidate any stale local copy). Encapsulated inside the header R/W functions — no caller changes.
  • Dependency: new optional C extension maru_shm._cxl_flush; if not built, Python falls back to a no-op with a one-time warning (single-host behavior unchanged).
sequenceDiagram
    participant A as Host A (RM / CLI)
    participant C as Host A CPU cache
    participant D as CXL device
    participant B as Host B (scanner)

    rect rgb(255, 235, 235)
    Note over A,B: Before — header trapped in Host A's cache
    A->>C: store header (mmap memcpy)
    C--xD: write-back deferred indefinitely (msync = no-op on DEV_DAX)
    B->>D: read header
    D-->>B: zeros → "no header" → B re-inits with a different UUID (split-brain)
    end

    rect rgb(235, 255, 235)
    Note over A,B: After — explicit cache-line flush
    A->>C: store header (mmap memcpy)
    A->>C: clflush + mfence
    C->>D: header line written back
    B->>B: clflush (drop stale local copy)
    B->>D: read header
    D-->>B: valid magic + UUID
    end
Loading

📝 Implementation Details

  • maru_resource_manager/src/device_header.cpp: flushCacheRange() (clflush per 64B line + mfence; x86-only, no-op elsewhere) — called after the store in writeDeviceHeader() and before the load in readDeviceHeader(). msync kept as best effort for regular-file backing (tests).
  • maru_shm/_cxl_flush.c (new): flush_range(buffer, offset=0, length=-1) via buffer protocol with bounds checking; HAVE_CLFLUSH constant. Registered in setup.py as an optional extension.
  • maru_shm/device_scanner.py: flush applied in write_device_header / clear_device_header (after store) and read_device_uuid (before load). mm.flush() wrapped in try/except — DEV_DAX may reject msync with EINVAL.

✅ Tests

  • Unit tests — 10 new in tests/unit/test_device_scanner.py (extension-missing fallback roundtrip, one-time warning, msync error swallow, flush_range bounds); C++ maru_rm_tests 16/16
  • Integration tests — tests/integration/test_daxmapping_flow.py passes
  • Manual tests — cross-host visibility verified on the multi-host window-alias harness: control 50/50, aliased writer 50/50 / reader 50/50, RM auto-init header identical through both windows

🔗 Related Issues (optional)

📦 Release Note (for auto-generation / write in English)

NEW

CHANGED

FIXED

  • RM: Device UUID header reads/writes now flush CPU cache lines so the header is visible and fresh across hosts sharing a CXL device (multi-host UUID resolution).

IMPORTANT NOTES

The RM UUID header is written and read with plain CPU load/store through
a DEV_DAX mmap. Those accesses go through the write-back CPU cache, and
msync() does not touch CPU caches on DEV_DAX (no page cache, no fsync
op). CXL 2.x shared memory has no cross-host cache coherence, so on a
multi-host setup a freshly written header stays invisible to other
hosts (both RMs auto-initialize -> UUID split-brain), and a previously
read header can go stale in the reader's cache when another host
rewrites it.

Fix per the multi-host simulation findings (CPU-store path stale 0/50
on every platform, corrected only by clflush+fence):

- C++ RM (device_header.cpp): clflush+mfence after the header store in
  writeDeviceHeader, and before the header load in readDeviceHeader.
- Python (device_scanner.py): same via a new maru_shm._cxl_flush C
  extension (flush_range), applied in write_device_header,
  clear_device_header and read_device_uuid. Falls back to a no-op with
  a one-time warning if the extension is not built, and msync errors
  (EINVAL on DEV_DAX) are now swallowed.
@github-actions

Copy link
Copy Markdown

@seohui-XCENA
seohui-XCENA marked this pull request as ready for review July 23, 2026 06:36
@seohui-XCENA
seohui-XCENA requested a review from a team July 23, 2026 06:37

@jooho-XCENA jooho-XCENA left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

로직·경계처리·에러처리 모두 정확해서 approve합니다. 요약과 minor 코멘트 몇 개 남깁니다.

정확성 확인

  • x86 메모리 모델 관점에서 올바름: CLFLUSH는 같은 캐시라인에 대한 이전 store와 순서가 보장되므로 writer 쪽 store → clflush → mfence, reader 쪽 clflush → mfence → load 모두 정확합니다.
  • 캐시라인 정렬 루프(addr & ~63 시작)가 경계 걸침을 올바르게 처리합니다.
  • flush_range의 bounds check가 offset > len 케이스(음수 length로 환원)까지 잡고, y*가 read-only 버퍼(PROT_READ mmap)를 수용하므로 read 경로도 문제없습니다.
  • msync를 best-effort로 강등한 것(DEV_DAX EINVAL)도 타당하며 회귀 위험 없습니다.
  • 테스트 커버리지(fallback roundtrip, 1회 경고, msync swallow, bounds) 좋습니다.

Minor

  1. 비-x86에서 경고 비대칭: ARM 등에서는 확장이 빌드되고 HAVE_CLFLUSH=0인데 flush_range가 조용히 no-op라서, _flush_range is None fallback 경고 경로를 타지 않아 경고가 전혀 안 나옵니다. device_scanner.py에서 _cxl_flush.HAVE_CLFLUSH == 0일 때도 같은 1회 경고를 내면 일관적입니다 (상수를 export해놓고 현재 사용처가 없음).
  2. clflushopt 대신 clflush를 택한 이유(가용성 우선, 헤더 32B라 성능 무의미)를 주석 한 줄로 남기면 좋겠습니다.
  3. 이 PR은 UUID 헤더만 flush합니다. RM 풀 메타데이터 등 다른 CPU-store 공유 구조가 같은 문제를 가질 수 있으니 후속 과제로 기록해두면 좋겠습니다.
  4. Extension(optional=True)는 최신 setuptools에서 deprecation 논의가 있는 필드입니다. 지금은 동작하니 blocker 아님.

멀티호스트 window-alias 하드웨어 검증을 머지 전 필수 게이트로 볼지 여부만 합의하면 될 것 같습니다.

jooho-XCENA and others added 2 commits July 23, 2026 17:12
- device_scanner: the one-time warning now also fires when the
  _cxl_flush extension is built without clflush support
  (HAVE_CLFLUSH=0, non-x86) — previously only a missing extension
  warned, so unsupported architectures silently skipped the flush.
- device_header.cpp / _cxl_flush.c: comment why clflush is used
  instead of clflushopt/clwb.
@seohui-XCENA

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review!

  • (1) Fixed in 81a0f6a — the one-time warning now also fires when the extension is built without clflush support (HAVE_CLFLUSH=0, non-x86), with a test.
  • (2) Added the clflush-vs-clflushopt rationale as comments in both flush helpers (same commit).
  • (3) Checked: the UUID header is currently the only CPU-store path to the CXL region — RM metadata (WAL/free list) lives on local disk and flows over TCP, and the KV data plane is GPU DMA only. Agreed it's worth watching whenever a new CPU-store path is added.
  • (4) Agreed, leaving optional=True as is.

Re: the merge gate — hardware verification is done. On the multi-host window-alias setup: control 50/50; aliased windows writer 50/50, reader 50/50; and the RM auto-init header reads back identically through both windows (covers the C++ write-flush and read-invalidate paths as well as the Python ones).

@seohui-XCENA
seohui-XCENA merged commit 5364e5a into xcena-dev:main Jul 23, 2026
3 checks passed
@seohui-XCENA
seohui-XCENA deleted the fix/device-header-cache-flush branch July 23, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants