fix(st): publish the lazy shard-mapping table without a data race - #1549
Merged
Merged
Conversation
…OTICE The LICENSE file still carried the Apache 2.0 template placeholders (Copyright [yyyy] [name of copyright owner]), so the notice was never actually applied to this work. Fill it in, add the NOTICE file Apache 2.0 expects for the copyright line, and point the four READMEs at both.
license: name the copyright holder and add NOTICE
st_shard_mapped() wrote g_st_shard_base/_len/_tried with no lock, no atomics and no barrier, and glm53.c reaches it from inside a `#pragma omp parallel for` (the batch loop at glm53.c:1587 -> expert_read -> st_map_shard_range). qwen38_core.h reaches it from a parallel region too. Two threads that both observe g_st_shard_tried[fd] == 0 both map the file, so one mapping is leaked for the process lifetime; and because the base pointer was published before the length, a third thread could take the non-NULL base and still read g_st_shard_len[fd] as 0, at which point st_map_shard_range rejects every range for that shard. Claim each fd once with a compare-exchange on g_st_shard_tried, store the length before release-storing the base, and read the base with acquire. A thread that loses the claim does not block: NULL is the documented "use pread" answer, so it takes the pread path for that one call instead of parking an OpenMP worker behind an mmap. Lock-free rather than the pthread mutex qwen38_core.h uses for its lazy HITS table, because st.h is also compiled into c/tools/check_glm53_container.c, whose documented build line is plain `gcc -O2 -std=gnu11 -Ic ... -lm` with neither -pthread nor -fopenmp. Latent today because COLI_MAP_EXPERTS is opt-in; JustVugg#1350 proposes flipping that default, which would make it live for everyone. Closes JustVugg#1519.
Edo771977
added a commit
to Edo771977/colibri
that referenced
this pull request
Sep 16, 2026
Integrazione dev (riuso KV Qwen3.6 JustVugg#1553) + JustVugg#1552 Qwen3.8 + JustVugg#1557 + JustVugg#1549
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1519.
The race
st_shard_mapped()wroteg_st_shard_base/g_st_shard_len/g_st_shard_triedwith no lock, no atomics and no barrier. It is reached concurrently:glm53.c:1587'scalls
expert_read→st_map_shard_range→st_shard_mappedfrom every worker at once, andqwen38_core.h:1102-1104does the same.expert_readalready knows it runs in parallel — it guardsm->misswith#pragma omp atomic— but the table it reaches through was unguarded.Two distinct failures follow:
g_st_shard_tried[fd] == 0, both map the file. One base pointer overwrites the other and that mapping is never unmapped — and shard mappings are deliberately kept for the process lifetime, so it is never reclaimed.g_st_shard_len[fd]as0.st_map_shard_rangethen failsoff > g_st_shard_len[fd] - nbytesfor every range on that shard, silently demoting it to pread for the rest of the run.Reproduction
TSan will not run on my host (aarch64, 47-bit VMA; TSan supports 39/42/48), so I reproduced the consequences directly instead. The harness copies the three globals and the lookup body verbatim out of
st.hand drives them from the same#pragma omp parallel for schedule(dynamic, 1)shape, counting how many times the map actually happens:Both failure modes appear on
mainand neither survives the patch. Happy to add the harness to the tree as a test if you'd like it — I left it out because it is a copy of the function body rather than a test of the real one.The fix
g_st_shard_tried.NULLis the documented "use pread" answer, so it takes the pread path for that one call rather than parking an OpenMP worker behind anmmap.Why lock-free and not the mutex used elsewhere
qwen38_core.hsolves the identical "lazy table published from a parallel region" problem withpthread_mutex_t+ double-checked locking + acquire/release — after, by its own comment, a SIGSEGV "in about one run in twelve on a CUDA warmstart". Copying that idiom was my first instinct.It does not work here:
st.his also compiled intoc/tools/check_glm53_container.c, whose documented build line in its own header iswith neither
-pthreadnor-fopenmp. Adding#include <pthread.h>tost.hwould silently invalidate that. The__atomic_*builtins are already used ~49 times in the tree and need no new dependency.Verification
gcc -O2 -std=gnu11 -Ic ... -lm(the tool's documented line, no pthread/OpenMP)gcc -fopenmp -pthread -c glm53.cqwen38.c,olmoe.c,qwen36.csyntax checkst.hstandalone with-Wall -WextraBehaviour is unchanged on the fast path: once a base is published, the lookup is a single acquire load, as before.
Note, not fixed here
st_map_experts_enabled()has the same shape (static int on = -1;initialised on first call) and is technically a data race under the memory model, though a benign one — every thread computes the same value from the samegetenv. I left it alone to keep this diff to the reported bug; say the word and I'll fold it in.