arena: allocate the abandoned-page bitmaps on first abandon; cheaper NUMA and option init - #32
Open
robobun wants to merge 3 commits into
Open
Conversation
Every (heap, arena) pair carried MI_ARENA_BIN_COUNT abandoned-page bitmaps next to its `pages` bitmap, laid out and initialized up front: once in the arena's info slices for the main heap, and once more in a zeroed allocation for every other heap that touched the arena (`mi_heap_ensure_arena_pages`). Each bitmap is sized by the arena (2 KiB per GiB of arena), so with the default 1 GiB reservation that is ~110 KiB per heap, and ~410 KiB for a heap in a 4 GiB arena. Writing the header of each bitmap touched one OS page per bitmap, so a heap paid about 50 page faults (or the memset, for external memory that is not known to be zero) before its first allocation, and most heaps never abandon a page at all: short-lived heaps are destroyed, not abandoned. Bun creates one heap per transpile and one for JSC's 4 GiB structure heap, so this was about 250 of the ~1000 page faults of `bun file.js`. Now `pages_abandoned[bin]` starts NULL and is allocated the first time a page of that bin is abandoned, from the subproc meta-data heap (safe on the abandon paths, where a regular heap allocation is not), published with a CAS. Readers treat NULL as all-clear; they were already guarded by `heap->abandoned_count[bin]`. `_mi_arena_pages_free` frees them with the heap's arena pages. If the allocation fails the page is abandoned unmapped, like a full page, and mapped once a free brings it back. The new test-abandoned-lazy exercises the abandon, reclaim, re-abandon, visit and delete paths from several threads at once.
…line Probing /sys/devices/system/node/nodeN with access() costs five or more syscalls at process start on a single-node machine (the loop allows four missing nodes before it stops), seven on a two-node one. The kernel lists the online nodes in one file; read it and fall back to the probe only when it is unavailable.
…riable is set _mi_options_init initializes every option at process load, and each one scanned the whole environment with a case-insensitive compare for mimalloc_<name> (and again for a legacy name). That is ~50 scans, each touching the option's name string in .rodata, for a process that sets no mimalloc variable, which is nearly all of them. A new prim, _mi_prim_getenv_has_prefix, answers once whether any variable starts with mimalloc_ (the unix prim walks environ; the others return unknown, which keeps the lookups). When the answer is no, mi_option_init takes the default without a lookup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
MI_ARENA_BIN_COUNT(49) abandoned-page bitmaps up front, each sized by the arena. For the default 1 GiB arena that is ~110 KiB per heap. For a heap in a 4 GiB arena it is ~410 KiB. The header write of each bitmap touches its own OS page, so a heap pays ~50 page faults, or a memset of that size for external memory that is not known to be zero, before its first allocation. Most heaps never abandon a page: a short-lived heap is destroyed.mi_manage_os_memory_exwithis_zero=false): about 250 of the ~1000 page faults of abun file.jsstart._mi_options_initand the NUMA probe add ~60 syscalls and environment scans to the same start.Fix
pages_abandoned[bin]starts NULL and is allocated the first time a page of that bin is abandoned (mi_arena_pages_abandoned_ensure), from the subproc meta-data heap, published with a CAS. Readers treat NULL as all-clear. They were already guarded byheap->abandoned_count[bin]. If the allocation fails, the page is abandoned unmapped, like a full page, and mapped when a free brings it back._mi_arena_pages_freereleases the bitmaps with the heap's arena pages._mi_prim_numa_node_counton Linux reads/sys/devices/system/node/online(one open, read, close) instead of oneaccess()per candidate node directory. It falls back to the probe when the file is unavailable.mi_option_initasks a new prim,_mi_prim_getenv_has_prefix("mimalloc_"), once. When no such variable exists, every option takes its default without scanning the environment. The unix prim walksenviron. The windows, wasi and emscripten prims return unknown, which keeps the lookups.test/test-abandoned-lazy.c(new: abandon from 8 threads at once, reclaim, re-abandon by a foreign free, delete, for a user heap and the main heap). ctest Debug (MI_DEBUG_FULL): 25/25. ctest Release withMI_USE_CXX: 24/24. The new test also passes under ASAN. I could not run the TSAN configuration: its runtime needs lower ASLR entropy, and the container blockspersonality, so every test stops beforemainwith or without this change.Background
heap->abandoned_count[bin]says whether any exist._mi_meta_zalloc_aligned) is the allocator mimalloc uses for thread-local data. It is usable on the abandon paths (a thread tearing down its theaps, a foreign free re-abandoning a page), where allocating from a regular heap is not.Measured in bun (release, x64 linux, interleaved A/B, 400 rounds,
bun empty.js): minor page faults 1035 to 785, wall 3.94 ms to 3.29 ms median, peak RSS 10.46 MB to 9.43 MB. Part of that is from two bun-side changes that ship with the bump.