fix: flush CPU cache lines in device UUID header read/write paths#67
Merged
seohui-XCENA merged 3 commits intoJul 23, 2026
Merged
Conversation
The RM UUID header is written and read with plain CPU load/store through a DEV_DAX mmap. Those accesses go through the write-back CPU cache, and msync() does not touch CPU caches on DEV_DAX (no page cache, no fsync op). CXL 2.x shared memory has no cross-host cache coherence, so on a multi-host setup a freshly written header stays invisible to other hosts (both RMs auto-initialize -> UUID split-brain), and a previously read header can go stale in the reader's cache when another host rewrites it. Fix per the multi-host simulation findings (CPU-store path stale 0/50 on every platform, corrected only by clflush+fence): - C++ RM (device_header.cpp): clflush+mfence after the header store in writeDeviceHeader, and before the header load in readDeviceHeader. - Python (device_scanner.py): same via a new maru_shm._cxl_flush C extension (flush_range), applied in write_device_header, clear_device_header and read_device_uuid. Falls back to a no-op with a one-time warning if the extension is not built, and msync errors (EINVAL on DEV_DAX) are now swallowed.
seohui-XCENA
marked this pull request as ready for review
July 23, 2026 06:36
jooho-XCENA
approved these changes
Jul 23, 2026
jooho-XCENA
left a comment
Contributor
There was a problem hiding this comment.
로직·경계처리·에러처리 모두 정확해서 approve합니다. 요약과 minor 코멘트 몇 개 남깁니다.
정확성 확인
- x86 메모리 모델 관점에서 올바름: CLFLUSH는 같은 캐시라인에 대한 이전 store와 순서가 보장되므로 writer 쪽
store → clflush → mfence, reader 쪽clflush → mfence → load모두 정확합니다. - 캐시라인 정렬 루프(
addr & ~63시작)가 경계 걸침을 올바르게 처리합니다. flush_range의 bounds check가 offset > len 케이스(음수 length로 환원)까지 잡고,y*가 read-only 버퍼(PROT_READ mmap)를 수용하므로 read 경로도 문제없습니다.- msync를 best-effort로 강등한 것(DEV_DAX EINVAL)도 타당하며 회귀 위험 없습니다.
- 테스트 커버리지(fallback roundtrip, 1회 경고, msync swallow, bounds) 좋습니다.
Minor
- 비-x86에서 경고 비대칭: ARM 등에서는 확장이 빌드되고
HAVE_CLFLUSH=0인데flush_range가 조용히 no-op라서,_flush_range is Nonefallback 경고 경로를 타지 않아 경고가 전혀 안 나옵니다.device_scanner.py에서_cxl_flush.HAVE_CLFLUSH == 0일 때도 같은 1회 경고를 내면 일관적입니다 (상수를 export해놓고 현재 사용처가 없음). clflushopt대신clflush를 택한 이유(가용성 우선, 헤더 32B라 성능 무의미)를 주석 한 줄로 남기면 좋겠습니다.- 이 PR은 UUID 헤더만 flush합니다. RM 풀 메타데이터 등 다른 CPU-store 공유 구조가 같은 문제를 가질 수 있으니 후속 과제로 기록해두면 좋겠습니다.
Extension(optional=True)는 최신 setuptools에서 deprecation 논의가 있는 필드입니다. 지금은 동작하니 blocker 아님.
멀티호스트 window-alias 하드웨어 검증을 머지 전 필수 게이트로 볼지 여부만 합의하면 될 것 같습니다.
- device_scanner: the one-time warning now also fires when the _cxl_flush extension is built without clflush support (HAVE_CLFLUSH=0, non-x86) — previously only a missing extension warned, so unsupported architectures silently skipped the flush. - device_header.cpp / _cxl_flush.c: comment why clflush is used instead of clflushopt/clwb.
Collaborator
Author
|
Thanks for the thorough review!
Re: the merge gate — hardware verification is done. On the multi-host window-alias setup: control 50/50; aliased windows writer 50/50, reader 50/50; and the RM auto-init header reads back identically through both windows (covers the C++ write-flush and read-invalidate paths as well as the Python ones). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤔 Background & Motivation (Why)
msync()does not flush CPU caches on DEV_DAX (no page cache, no fsync op).clflush+fence.🏗️ Design Changes
clflush+mfence(write-back to device), header reads now start withclflush+mfence(invalidate any stale local copy). Encapsulated inside the header R/W functions — no caller changes.maru_shm._cxl_flush; if not built, Python falls back to a no-op with a one-time warning (single-host behavior unchanged).sequenceDiagram participant A as Host A (RM / CLI) participant C as Host A CPU cache participant D as CXL device participant B as Host B (scanner) rect rgb(255, 235, 235) Note over A,B: Before — header trapped in Host A's cache A->>C: store header (mmap memcpy) C--xD: write-back deferred indefinitely (msync = no-op on DEV_DAX) B->>D: read header D-->>B: zeros → "no header" → B re-inits with a different UUID (split-brain) end rect rgb(235, 255, 235) Note over A,B: After — explicit cache-line flush A->>C: store header (mmap memcpy) A->>C: clflush + mfence C->>D: header line written back B->>B: clflush (drop stale local copy) B->>D: read header D-->>B: valid magic + UUID end📝 Implementation Details
maru_resource_manager/src/device_header.cpp:flushCacheRange()(clflushper 64B line +mfence; x86-only, no-op elsewhere) — called after the store inwriteDeviceHeader()and before the load inreadDeviceHeader().msynckept as best effort for regular-file backing (tests).maru_shm/_cxl_flush.c(new):flush_range(buffer, offset=0, length=-1)via buffer protocol with bounds checking;HAVE_CLFLUSHconstant. Registered insetup.pyas an optional extension.maru_shm/device_scanner.py: flush applied inwrite_device_header/clear_device_header(after store) andread_device_uuid(before load).mm.flush()wrapped in try/except — DEV_DAX may reject msync with EINVAL.✅ Tests
tests/unit/test_device_scanner.py(extension-missing fallback roundtrip, one-time warning, msync error swallow,flush_rangebounds); C++maru_rm_tests16/16tests/integration/test_daxmapping_flow.pypasses🔗 Related Issues (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
FIXED
IMPORTANT NOTES