Heap delete/destroy: one teardown protocol (supersedes #22–#26) - #27
Merged
Conversation
…als were freed Upstream 932f6e5 (the threadlocal.c part only; the page-map commit policy, MI_MAX_EXTEND_SIZE and MI_INIT_PAGES_DIRECT riders of that commit are not taken here).
(cherry picked from commit a15a51d)
…index The scan starts at node1, so on a machine with nodes 0..N-1 `last_found` ends at N-1 and is returned as the count: 2 nodes reported as 1, 4 as 3. Callers use it as a bound on node ids (`numa_node % numa_count`), so the highest node gets folded onto node 0. Return `last_found + 1`, as the Windows primitive does with `GetNumaHighestNodeNumber() + 1`. Follow-up to 870ac85. (cherry picked from commit 66383f0)
…hread has no theap for its heap (upstream issue microsoft#1364) (cherry picked from commit 53b364d; the fork's _mi_theap_can_touch assertions are kept where the scavenger acts for a parked thread)
With allow_thp off, unix_mmap called madvise(MADV_NOHUGEPAGE) on every mapping it created. The kernel only backs a mapping that did not ask for huge pages with them when /sys/kernel/mm/transparent_hugepage/enabled is [always]. Under [madvise] (the Debian and Ubuntu default) and [never] the call changes nothing and costs one syscall per mmap: three during process initialization and one per arena after that. unix_detect_thp already reads that file. It now also records whether the setting is [always], and unix_mmap makes the madvise call only in that case. When the file cannot be read (no sysfs), the opt-out stays in place. The per-size mTHP settings are not read: that would take more syscalls than it saves. test-thp-optout interposes madvise, counts the MADV_NOHUGEPAGE calls the allocator makes during initialization (the ctest entry runs it with MIMALLOC_ALLOW_THP=0) and for a fresh reservation, and checks them against the setting of the machine it runs on. With allow_thp on it expects none.
…pted link A free block holds its next link in its own first word. When something writes into a block after it was freed, or a stale free links a live object, the link points anywhere. Release builds do not encode links, so the walks that read a whole list followed it and faulted: the idle sweep (mi_page_purge_holes_walk), the forced collect of local_free in mi_page_free_collect_ex, and mi_page_thread_collect_to_local, which bounded the length of the thread-free list but not where its links point. This is the crash family behind oven-sh/bun BUN-40BH, BUN-40CP, BUN-40SQ and BUN-41H5. The three walks now check every link: it has to be NULL or the start of a formed block of the page. The sweep also tracks which blocks it has seen, so a block listed twice is found instead of counted twice (two counts for one block could make an OS page with one live block look entirely free and get discarded) and a cyclic list ends. A corrupted list is reported through _mi_error_message (EFAULT, as the existing thread-free message does) and cut in front of the block holding the bad link; the blocks cut off are counted as used from then on, so they are never handed out and the page is never returned to the arena while one of them may still be live. The sweep knows how many blocks remain listed and sets used exactly; the collects bump it by one. The thread-free collect keeps its existing recovery and drops the list it took. The scavenger thread blocked every signal, including the ones a fault on the thread itself raises. A blocked SIGSEGV or SIGBUS is not queued: the kernel resets it to the default action and kills the process, so a fault during a sweep the scavenger did for a parked thread ended the process with no report from the host's crash handler (verified on Linux). Leave the thread-directed fault signals unblocked; the process-directed ones stay blocked as before. test-freelist-corruption scribbles a freed block the way the crashes look (a link 8 bytes into the block, the word 0xA0D behind it) on each of the three lists, plus a bad head and a self-linked block, and checks the report, the cut, the block accounting, that a second pass is quiet, that the block is not handed out again and that the live blocks are intact. Without the src changes four of the five tests fault and the self-link test loops.
…oncurrent mi_heap_delete can free the heap mi_heap_delete claims each page of the heap, re-points page->heap to the main heap, and then frees the heap struct. A mi_free from another thread runs concurrently with that. Two places on its path read page->heap at a point where the deleter does not wait for them: - mi_stat_free (MI_STAT>0) read page->heap->subproc before it owns the page. The heap struct can be freed between the two loads, and the free list link is written over heap->subproc (its first field), so the next load is garbage and subproc->theap_meta faults. This is the SIGSEGV in test-heap-mt (heap-free-during-delete-overlap) under load. The stats now find the subproc through the page's arena, which is constant for the lifetime of the page (and through the current subproc for an OS allocated page), and the meta theap test compares the page's thread id with MI_THREADID_DETACHED directly. - mi_arenas_page_free_prim read page->heap->subproc after it cleared the page's bit in the heap's arena_pages. That bit is what the deleter waits on, so the heap can be gone by then. Read the subproc once, before the bit is cleared. Document the rule on mi_page_heap and on the page->heap field.
…e is freed mi_bitmap_clear updates the chunkmap of the bitmap after it clears the bit. The bit is what a concurrent mi_heap_delete waits on before it frees the heap and its arena_pages (mi_heap_visit_page_at), so that chunkmap update could touch a freed arena_pages. Add mi_bitmap_clear_no_chunkmap, which leaves the chunkmap bit set (allowed: it means the chunk may have bits set, and the pages bitmap is only ever visited, never searched), and use it in mi_arenas_page_free_prim. Narrow the comment on mi_page_heap to what the delete actually waits for.
…em after mi_heap_delete (and mi_heap_destroy) freed the theaps of the heap before it walked the pages, and the detach marked a theap by clearing theap->tld. A thread that used the heap before still reaches its theap for the heap through the heap's thread local while it frees blocks of the heap during the delete (_mi_page_associated_theap_peek on the reclaim and re-abandon paths of mi_free_try_collect_mt). That theap is then detached, with a NULL tld that the callers dereference, or already freed. Now the detach clears theap->heap instead (as _mi_tld_detach_theaps does on thread exit), which is what the peek compares, so a free that runs after the detach does not get the theap anymore. The theap struct and its tld stay valid until after the page walk, for a free that obtained the theap just before the detach: the walk waits for such a free while it owns the page. The statistics are still merged before the walk, so the page statistics of the heap are unchanged. _mi_heap_theap_peek returns NULL for a detached theap as well instead of asserting. test-heap-mt gets a variant where the pages are abandoned (the allocating thread exited) and every freer allocated from the heap once. It crashed in the first iteration before (mi_theap_matches_thread through _mi_arenas_page_try_reabandon_to_mapped).
Armed after pthread_create, the thread can have exited before the store and then never parks, and the wait for it spins forever. Seen once as a 1500s ctest timeout under a loaded ctest -j4.
…ap address Every thread keeps the theap of the heap it last used in `_mi_theap_cached`, and `_mi_heap_theap` takes that cache whenever `theap->heap` equals the requested heap. Upstream 36e8fc3 ("improve concurrent heap_delete and thread termination") dropped the `theap->heap = NULL` store from the heap-delete detach path, which the old `_mi_theap_free` had with a comment that it avoids an ABA where the cache holds a heap address that a newly allocated heap reuses. The previous commit restores that store (under the tld lock, as the thread-exit path does). This test pins it: B: H = mi_heap_new() A: mi_heap_malloc(H) -> A caches its theap for H B: mi_heap_destroy(H); H2 = mi_heap_new() -> H2 == H (LIFO reuse of the block) A: mi_heap_malloc(H2) -> must come from a theap of H2 Without the store a debug build segfaults in mi_heap_malloc (the cached theap is detached, `tld == NULL`) and a release build hands out 127 of 1024 blocks that `mi_heap_of` attributes to no heap: memory that went back to the arena. With the store every block belongs to H2.
A heap is torn down while other threads may still free blocks of it, and while threads that used it earlier still hold a theap for it in their thread-locals. That was handled reader by reader (do not read page->heap here, re-order this free there, restore that NULL store) and one hole was left in the middle: the deleter claimed pages by writing the owned bit of a page that nothing pinned, so a concurrent free could release the page and the slice be reused between the deleter's bitmap read and its write. The teardown is now the same four steps for delete and destroy (heap.c): 1. detach every theap of the heap from its thread and clear theap->heap, so no thread finds a theap for the heap anymore (allocation, reclaim on free, abandon on thread exit all go through that field); 2. abandon the pages of each detached theap, as thread exit does, so every page of the heap is an abandoned page and the only other party that can hold one is a mi_free collecting it -- this also picks up theap-held OS pages, which were not visited at all before; 3. claim each abandoned page and move or free it. A page is pinned before it is written to: its `arena_pages->pages` bit is cleared while set (the owner side, mi_arenas_page_free_prim, unpublishes with mi_bitmap_clear_once_set and so waits out the pin), or, for an OS page, the os_abandoned_pages lock is held. If the page is owned by a concurrent free the pin is dropped and the claim retried; before, an owned OS page was skipped and left on the list of the freed heap. This is the protocol the abandoned-page map already uses for allocation and the hole sweep. 4. free the theap structs, then the heap. mi_bitmap_clear_no_chunkmap goes away again (clear_once_set does not touch the chunkmap either). The contract is written down at the top of the section in heap.c: threads that never allocated from the heap may free into it during a delete; a thread that did must not use it during the delete, and the deleter now reports it if one does instead of silently taking a page out from under it.
…after it is unpublished test/test-heap-teardown.c covers the delete/destroy contract next to test-heap-mt: - pin (debug): the deleter holds a page pinned but not claimed while a concurrent free empties it; the free may not release the page until the pin is dropped, and the deleter must then find it gone. With the same hook in the old claim loop this fails 4 of 4. - os-pages: over-aligned (OS allocated) blocks held by the deleting thread's theap, by an exited thread, and freed by other threads during the delete. The first crashed before (the page was never visited and kept pointing at the freed heap), the last was a use-after-free found by TSAN: the free unlinked the page from the heap's list and then read page->heap for the subproc and statistics. _mi_arenas_page_free now resolves what it needs from the heap first and mi_arenas_page_free_prim takes it as arguments; freeing an abandoned page goes through _mi_arenas_abandoned_page_free which un-abandons (the unpublish, for an OS page) in between. - foreign-theap: a live thread holds arena and OS pages of the heap in its theap while another thread deletes it, frees afterwards, gets a new heap at the same address, and exits with the stale theap still cached. - page-churn, two-deletes, parked: stress for frees that empty pages during a delete with slice reuse pressure, concurrent deletes, and a parked thread whose theaps the scavenger sweeps while its heap is deleted. _mi_theap_can_touch checks `heap == NULL` before it looks at the tld: the thread of a heap-detached theap may have terminated (test-heap-delete-race under ASAN). Also: test-purge-holes `unformed-tail` compared against the requested block size instead of the padded one and so always failed in debug builds; the white-box tests are only built where they can link (not against the shared library under TSAN, freelist-corruption not under ASAN); mi-heapview links the sanitizer runtime; CI runs for bun* branches and their pull requests.
…g-field-initializers)
This was referenced Aug 23, 2026
…'s theap for it; tests: atomics for the heap-aba / heap-delete-race handshakes The theap returned by _mi_page_associated_theap_peek was only used for its (non-atomic) statistics on the re-abandon path, and a concurrent heap delete may be detaching that theap and merging those statistics (TSAN, test-heap-mt heap-free-during-delete-abandoned). The two tests synchronized through volatile ints, which TSAN cannot see through now that the deleter walks the other thread's theap.
(cherry picked from commit 0424ab2)
…ng or snapshotting; debug test hooks are atomics _mi_stats_print printed heap/subproc statistics while other threads (the scavenger's purge) update them with atomic adds; it now prints a copy taken with the same atomic loads the merge uses. mi_heap_snapshot memcpy'd the live arena bitmaps; it reads them a field at a time. The two MI_DEBUG test hooks were volatile ints written and read across threads. test-fork-user-heap runs with TSAN_OPTIONS=die_after_fork=0 under TSAN (the child starts the scavenger). All found by the TSAN configuration of the test suite, which CI now runs.
…rked A second _start without an _end wrote park_theap0/park_reclaim/park_swept while the scavenger, which had claimed the first park, was reading them (TSAN, test-park-handoff `unbalanced`). Only the owner leaves RUNNING, so checking for RUNNING first makes the stores race-free.
…s not SHA-pinned, which the org policy rejects for the whole workflow)
…iguration These failed on bun-dev3-v2 already with plain `cmake -DCMAKE_BUILD_TYPE=Release` (the CI configuration, which had never run on this branch): - prof.c did not compile without -D_GNU_SOURCE (struct dl_phdr_info); static.c and prof.c define it on Linux before the first libc header. - test-prof-adversarial: with a sample rate of 1, glibc's backtrace() dlopen's libgcc_s on first use and that dlopen callocs before it is re-entrant, so the sampler sampled its own allocation without end (stack overflow in ld.so). _mi_prof_sample does not sample allocations made while it takes a backtrace (tld->prof_sampling). - test-stress-subprocs: mi_process_done pthread_join'ed the scavenger from the exit handler; glibc trims its thread stack cache inside a join and frees the DTVs of long dead threads there, which were malloc'ed (by us, overridden) on a sub-process thread into an arena that mi_subproc_destroy had unmapped. At process exit we now only wait for the scavenger to leave the allocator (_mi_scavenger_stop); the public mi_scavenger_stop still joins (_mi_scavenger_stop_and_join). - test-purge-holes asserted process-wide hole counters are zero, which does not hold when the C runtime's own allocations go through mimalloc; it compares deltas for its own page and skips the global at-exit check when malloc is mimalloc.
…roc_destroy when malloc is overridden Not joining (or detaching) the scavenger at exit only moved glibc's stack-cache trim around. The memory it trips over is the DTVs glibc allocated on the sub-process threads for the worker threads they created, which mi_subproc_destroy unmaps with the rest of the sub-process: destroying a sub-process whose threads made C-library-internal allocations is not valid under malloc override, and the test now says so and skips the destroy in that configuration. mi_process_done joins the scavenger as before.
…static The workflow never ran on this fork's branches, so the fork's own tests and a few library bits had only ever been built as C on Linux: library - scavenger: start on first demand (a purge is scheduled, or a thread parks) instead of in mi_process_init. As an inserted dylib on macOS our initializer runs before libobjc's, which aborts if a thread already exists (test-stress-dynamic: "task_restartable_ranges_register failed"); and a short-lived process no longer gets a thread it never uses. - the MI_DEBUG-only assertions in mi_theap_is_valid / _mi_page_is_valid compared the validating thread's own theap for the heap with the page's; that does not hold on the scavenger while it collects a parked thread's theaps (FreeBSD). - the test hooks (mi_debug_stall_*, mi_debug_fail_os_commit_after) are declared with C linkage so C tests link against a library compiled as C++ (MSVC). - os.c: no --/assignment-result on a volatile (C++20 deprecation, bun builds this file as C++). tests / cmake - white-box tests are compiled as C++ when the library is (they call internal, now mangled, functions); test-purge-holes made valid C++ (no goto across an initialization). - POSIX-only tests (theap-sentinel, park-handoff, heap-churn, prof-adversarial) are stubs on Windows like fork-user-heap already was; MSVC compiles the C tests with /experimental:c11atomics; test-emulated-tls only where the runtime has __emutls_get_address (Linux/Android); mi-heapview without __builtin_popcount; test-snapshot includes <process.h> for getpid. - test-purge-holes report-pinned-ospages: `pending` need not be 0 in mode 0 (formed blocks past the last one handed out; showed under MI_SECURE with purge_holes off); test-freelist-corruption local-free-link: the surviving block may have been discarded rather than left on `free` (MI_SECURE layout randomization made this fail ~1 in 6). - test-heap-mt: 1000 iterations in debug builds (the FreeBSD VM timed out).
…erations, C++ builds of the heap tests on MSVC; stats: mi_stats_copy reads live counters atomically; cmake: -save-temps=obj - test-purge-holes: 'sweep-skips-unchanged-pages' picks its live blocks by OS page instead of assuming 8 blocks of 512 bytes per 4 KiB OS page (Win32, 16 KiB pages); the mode-0/1 report checks only assert what holds for any OS page size and extension policy (macOS 16K, MI_SECURE) -- the exact numbers are already checked against the per-page ground truth. - test-heap-mt defaults to 1000 iterations (5000 timed out on the 3-core macOS runners at 240 s). - MSVC: the heap tests and the white-box tests compile as C++ like the library (C11 atomics, mangled internals); the debug hooks are extern "C" on the test side too. - mi_subproc_stats_get memcpy'd the live subproc stats (TSAN, test-stress); it copies per counter. - MI_SEE_ASM used -save-temps, whose temporaries collide between targets that compile the same sources (mimalloc and mimalloc-emulated-tls): the shared library got an alloc.o preprocessed for the other target's TLS model. -save-temps=obj keeps them per object directory.
…free), initialize its mutex/condvar eagerly; tests: guarded/32-bit/16K-page expectations - The lazy start moved to thread initialization (second thread onward) and mi_on_thread_idle_start; a purge that is due without a scavenger runs inline as before. Starting it from _mi_arenas_purge_now meant pthread_create (which allocates) from inside a page free. - The generic POSIX wait/wake used statically initialized pthread objects; FreeBSD's libthr allocates those on first use, which under malloc override is an allocation by a thread that has just published itself as parked, racing the sweep of its theaps (test-heap-teardown 'parked', test-park-handoff and test-stress-subprocs asserting on FreeBSD). They are initialized when the scavenger starts. - tests: purge-holes and freelist-corruption know the page layout and are not built with MI_GUARDED; prof-adversarial's realloc case reallocates to a size that cannot grow in place; large-pages accepts singleton pages (32-bit); sweep-skips picks live blocks per OS page.
…al layout (16 KiB pages, 32-bit); large-pages accepts singleton pages that the sweep cannot reach
…ut not under sanitizers The acquire load taken from upstream 2dc67ef sits on every _mi_ptr_page here (upstream moved the page map off that path with the aligned page meta), and under TSAN an acquire is a sync-map lookup: test-stress went from 145 s to 370 s and timed out in CI. The pointer being freed was obtained through synchronization with the allocating thread, which registered the page before handing it out, so the relaxed load that bun-dev3-v2 ships is kept. test-stress under TSAN runs 100 iterations like upstream; test-thp-optout counts madvise calls by interposing the symbol, which a sanitizer runtime does first, so it is not built there.
…assertion on the allocation failure path, reached with MI_GUARDED when mprotect fails)
…ng per block would exceed vm.max_map_count)
… static initializer for the scavenger's mutex/condvar mi_register_error_message registers a handler that receives the formatted text of an _mi_error_message as well as the error code, so that an embedder can put "corrupted free list in page P (block size N): invalid link X in block B" into its crash report instead of only learning that an EFAULT happened. test-freelist-corruption uses it. The scavenger's pthread mutex and condition variable were statically initialized and then initialized again with pthread_*_init in mi_scav_init, which POSIX leaves undefined; they are now only initialized there (every wait and wake is reached after _mi_scavenger_start). A heap deleted while a thread that allocated from it is still using it is a contract violation: assert in debug instead of printing, keep the best-effort seize in release.
…weep from acting on a list that does not check out This takes back the link validation and list cutting from a8dfced. The corrupted free lists in oven-sh/bun's crash reports (BUN-40BH and its siblings) are written by bun on Windows (a file read completing into a freed buffer, oven-sh/bun#39897, and a poll handle freed twice from a nested event loop, oven-sh/bun#39643); the allocator is where it shows, not where it happens, and repairing the list here would hide the writer. What stays from that change: the scavenger thread no longer blocks the fault signals, and the hole sweep does not discard memory based on a free list it cannot trust -- a block listed twice would be counted twice and make an OS page with a live block in it look free, and a link that is not a block of the page would index outside the walk's arrays -- it leaves such a page alone (and asserts in a debug build). mi_register_error_message is dropped again too.
…list, the scavenger's lazy start, and its signal mask test-sweep-double-free: a block freed twice puts a cycle in its page's free lists; the idle sweep must return and must not discard the OS page the live neighbour is in. On 6a14aee (release) the sweep never returns. test-park-handoff, first thing in main (Linux, reads /proc): a process that only ever had one thread has no mi-scavenger thread; the first park starts exactly one; its blocked mask has the process-directed signals but not SIGSEGV/SIGBUS. On 6a14aee the first and the last of these fail.
It is not free (a bit set per free block and a memzero per swept page) and it is handling for a bug that is the program's, not the allocator's. A debug build already refuses the second free in mi_free and asserts the block index in the walk. mi_page_purge_holes_walk is back to what bun-dev3-v2 has.
Jarred-Sumner
added a commit
to oven-sh/bun
that referenced
this pull request
Aug 23, 2026
No-Verification-Needed: dependency pin bump to the merge commit; tree identical to the previous pin
Jarred-Sumner
added a commit
to oven-sh/bun
that referenced
this pull request
Aug 24, 2026
) (#40138) ### What does this PR do? Bumps mimalloc to oven-sh/mimalloc#27, which combines oven-sh/mimalloc#22–#26 and replaces their per-reader fixes with one teardown protocol for `mi_heap_delete` / `mi_heap_destroy`. The problem those PRs were circling: a heap is torn down while a concurrent cross-thread `mi_free`, or a thread that used the heap earlier and still caches a theap for it, can reach it. The hole in the middle was that the deleter claimed pages by *writing* to them (`atomic_or` of the owned bit) with nothing pinning the page, so a concurrent free could release the page and the slice be reused in between. Now: detach theaps → abandon their pages as thread-exit does → pin-then-claim every abandoned page (the same bitmap-as-pin protocol the abandoned-page map already uses) → free theaps → free heap. Details, contract and tests in the mimalloc PR. Also in the bump: mimalloc#22 (THP opt-out only when the system setting is `always` — saves a `madvise` per mmap on Debian/Ubuntu defaults), from mimalloc#23 only the scavenger signal mask (a fault on that thread produced no crash report), the scavenger thread starting lazily (a single-threaded `bun -e` no longer spawns it: −1 thread, −24 syscalls), and targeted upstream dev3 fixes (thread-locals-after-free guard, #1364, #1371, NUMA node count). Upstream's in-progress page-meta layout rework is deliberately *not* included. **What this does not do:** fix the Windows corrupted-free-list crash family (BUN-40BH and siblings). Those lists are written by Bun — #39897 (file read completing into a freed buffer, merged) and #39643 (poll handle freed twice from a nested event loop, open) — and mimalloc is only where the damage surfaces. #23's "validate links and cut the list" is not taken for that reason: it would keep running past the write and hide it. ### How did you verify your code works? - mimalloc `ctest`: Release 23/23, Debug (`MI_DEBUG_FULL`) 24/24 (was 20/21 on the old pin), ASAN 22/22, TSAN 19/19 with 0 reports (see mimalloc#27). - `bun bd test`: transpiler (190/190), bundler_edgecase (138/138), bundler_minify (43/43), css (2358 pass; 6 debug-timeout fuzz tests), workers/serve (same 4 failures as a `main` debug build on this box). - Release x64, n=7 interleaved, old pin vs new pin on the same Bun commit: | | old pin median (range) | new pin median (range) | Δ | |---|---|---|---| | `bun -e 1` peak RSS | 27024 KB (26564–27088) | 26016 KB (25984–26020)¹ | −3.7% | | `bun -e 1` syscalls | 270 | 246 | −24 (no `clone3` for the scavenger, −6 `rt_sigprocmask`, −5 `madvise`) | | `Bun.serve` hello RSS after 200k req (c=64) | 49760 KB (48540–50040) | 47948 KB (46384–48252) | −3.6% | | `bun build --minify --sourcemap` three.js×10 peak | 345696 KB (340672–348928) | 343724 KB (339612–348828) | −0.6% (overlaps) | ¹ one run at 17212 KB excluded from the range as an outlier. The first two rows come from the scavenger thread now starting on first use (first park / first scheduled purge) instead of at process init — a change made because the eager start aborted macOS processes that `DYLD_INSERT` the dylib (thread created before libobjc initializes); `bun -e 1` never needs it. Before that change the same A/B was flat (+0.1–0.3%, overlapping), so the teardown protocol itself is RSS-neutral as forecast.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Combines and supersedes #22, #24, #25 and #26; takes two pieces of #23 and drops the rest.
What this is not: a fix for the corrupted free lists in Bun's crash reports (BUN-40BH and its ~40 siblings). Those lists are written by Bun on Windows — a file read completing into a freed buffer (oven-sh/bun#39897, merged) and a poll handle freed twice from a nested event loop (oven-sh/bun#39643) — and the allocator is only where it shows. Nothing here repairs or hides such a list.
The one problem behind #24 / #25 / #26
mi_heap_delete/mi_heap_destroytear a heap down while two other parties can still reach it: a concurrent cross-threadmi_freeinto one of its pages, and a thread that used the heap earlier and still has a theap for it in its thread-locals. That was being handled reader by reader (#24: don't readpage->heapinmi_stat_free/ after the bitmap clear; #25: detach order andtheap->heap = NULL; #26: a test for the cache ABA), and one hole sat in the middle of it:mi_page_claim_ownership, which is anatomic_oronpage->xthread_free— a write to a page that nothing pins. A concurrent free can release the page and the slice be reused between the deleter'sarena_pages->pagesbit read and that write, so the deleter flips a bit in memory that is no longer this heap's page (or "owns" a page of another heap and migrates it).mi_heap_delete(h); mi_free(huge_block)used the freed heap.page->heapfor the subproc and statistics (TSAN, with the new test).What this does instead
One teardown, the same for delete and destroy (
heap.c, with the contract written on top of it):theap->heap— no thread finds a theap for the heap anymore (allocation, reclaim-on-free and abandon-on-exit all go through that field);mi_freecollecting it;arena_pages->pagesbit is cleared while set (the owner side unpublishes withmi_bitmap_clear_once_set, so it waits out the pin), or for an OS page the list lock is held; if the page turns out to be owned the pin is dropped and the claim retried. This is the protocol the abandoned-page map already uses for allocation and for the hole sweep (mi_arena_try_claim_abandoned,mi_arena_page_purge_holes_at)._mi_arenas_page_freeresolves subproc /arena_pagesfrom the heap first andmi_arenas_page_free_primnever readspage->heap; an abandoned page is freed through_mi_arenas_abandoned_page_free, which un-abandons (the unpublish for an OS page) in between.mi_bitmap_clear_no_chunkmapfrom #24 goes away again (clear_once_setdoesn't touch the chunkmap either). If a thread that allocated from the heap is still using it during the delete (outside the contract) the deleter now says so instead of silently taking the page.Contract (top of the section in
heap.c): threads that never allocated from the heap maymi_freeinto it during a delete; a thread that did allocate from it must not use it during the delete and may free afterwards; thread exit may race freely.#23 and #22
From #23 only what is mimalloc's own: the scavenger thread no longer blocks the fault signals (a crash during a sweep it does for a parked thread produced no report at all). Nothing else: #23's link validation / list cutting in the three cold walks is not taken — it would keep a process running past a write into freed memory and hide the writer — and the sweep walk is unchanged from
bun-dev3-v2. #22 (THP opt-out only underalways) is unrelated and rides along.Upstream dev3
105 commits ahead of our last sync. Taken here, as separate commits: the thread-locals-after-free guard (932f6e5, threadlocal.c part only), re-abandon without a theap (microsoft#1364, 53b364d), realloc assertion (microsoft#1371), NUMA node count (66383f0), the osx
zone_freeassertion (b2ef742). (The acquire load of the page-map submap from 2dc67ef was tried and dropped again: here it sits on every_mi_ptr_pageand made the TSAN suite 2.5× slower; upstream only did it after moving the page map off that path.) Not taken: the aligned-page-meta layout rework and what sits on top of it (double-free-via-padding, page commit granularity, retire changes) — that is most of the 105, was still landing yesterday, touches the layout the hole sweep depends on, and should be its own sync PR rather than ride into Bun with a correctness fix.Also fixed along the way (CI had never run on this fork's branches)
The workflow only triggered on
dev*, so nothing below had ever been built outside Linux/C/static, and several tests failed onbun-dev3-v2already:mi_process_init— as an inserted dylib on macOS our initializer runs before libobjc's, which aborts if a thread already exists (test-stress-dynamic), and a single-threaded short process no longer gets a thread it never uses; its generic-POSIX mutex/condvar are initialized eagerly (FreeBSD allocates statically initialized ones on first use — an allocation by a thread that just published itself parked); a doublemi_on_thread_idle_startno longer writes park fields under the scavenger;prof.cneeds_GNU_SOURCE; the profiler sampler does not sample allocations made while taking a backtrace (glibcbacktrace()dlopen's libgcc_s → calloc → recursion at rate 1); stats print/get/snapshot read live atomically-updated counters/bitmaps with atomic loads; debug-onlymi_theap_is_valid/_mi_page_is_validasserts were owner-only and fired on the scavenger; test hooks are C-linkage atomics.test-purge-holeshad four expectations tied to 64-bit/4 KiB/no-padding geometry (one had failed in every debug run since it was written) and asserted process-wide counters that don't hold under malloc override;test-stress-subprocsmay notmi_subproc_destroyunder malloc override (glibc's cached thread DTVs live in that memory);-save-temps=objsoMI_SEE_ASMbuilds don't race on temporaries between targets; actions pinned per org policy, Alpine jobs dropped (unpinnable nested action),fail-fast: false.Tests
test/test-heap-teardown.c(new):pin(debug; deterministic via a hook at the pin/claim point — fails 4/4 with the same hook in the old claim loop),os-pages(crashed before),foreign-theap(crashed before with an OS page held),page-churn,two-deletes,parked.test-heap-mt,test-heap-aba(mi_heap_delete: detach the theaps before the pages are moved and free them after #25/test-heap-aba: regression test for the theap cache ABA on a reused heap address (stacked on #25) #26),test-thp-optout(unix: only opt mappings out of THP when the system setting is always #22).test-park-handoff(new first case, Linux): a single-threaded process has nomi-scavengerthread; the first park starts exactly one; its blocked mask contains the process-directed signals but not SIGSEGV/SIGBUS. On 6a14aee the first and last fail.heap-teardownpage-churn,two-deletes,parked.pinis debug-only (it needs the stall hook to open the window deterministically).test-purge-holesunformed-tailhas been failing in every debug run: it compared against the requested block size instead of the padded one. Fixed; debug ctest is 24/24 now (was 20/21).MI_DEBUG_FULL) 24/24, ASAN 22/22, TSAN 19/19 with 0 reports. Getting TSAN to zero meant fixing pre-existing races too:_mi_stats_printreading live atomically-updated counters with plain loads,mi_heap_snapshotmemcpy of live bitmaps, thevolatile intdebug hooks, a doublemi_on_thread_idle_startwriting park fields under the scavenger, and volatile-based handshakes in test-heap-aba / test-heap-delete-race. White-box tests are only built where they can link (TSAN links tests against the shared lib),mi-heapviewlinks the sanitizer runtime, test-fork-user-heap runs withdie_after_fork=0under TSAN.bun*branches and their PRs (it only triggered ondev*, so none of unix: only opt mappings out of THP when the system setting is always #22–test-heap-aba: regression test for the theap cache ABA on a reused heap address (stacked on #25) #26 ever ran it).