Linux (maybe macOS) self-deadlock in v3.3.0 with `MI_SECURE`>=4 on large aligned allocation

[chirizxc]

On Linuxx86_64 and it seems on macOS, mimalloc v3.3.0 hangs with MI_SECURE=4 and MI_SECURE=5 on a large aligned allocation.

MRE:

#include <stdio.h>
#include "mimalloc.h"

int main(void) {
  void* p = mi_malloc_aligned(1 << 20, 32);
  if (p == NULL) return 1;
  mi_free(p);
  return 0;
}

cc -Iinclude -DMI_SECURE=4 -o repro repro.c src/static.c -lpthread ./repro
# or cc -Iinclude -DMI_SECURE=5 -o repro repro.c src/static.c -lpthread ./repro

GDB shows a self-deadlock:

• mi_arenas_try_alloc holds subproc->arena_reserve_lock
• secure path in _mi_os_get_aligned_hint calls mi_theap_get_default()
• this re-enters mi_thread_init()
• thread init allocates metadata
• allocation re-enters mi_arenas_try_alloc
• same thread blocks again on arena_reserve_lock

Relevant frames:

• mi_lock_acquire in include/mimalloc/atomic.h
• mi_arenas_try_alloc in src/arena.c
• _mi_os_get_aligned_hint in src/os.c
• _mi_thread_init_theap_default / mi_thread_init in src/init.c

[chirizxc]

https://github.com/microsoft/mimalloc/blob/dev3/src/os.c#L130-L153

something like this should fix the problem:

--- a/src/os.c
+++ b/src/os.c
@@ -138,8 +138,14 @@ void* _mi_os_get_aligned_hint(size_t try_alignment, size_t size)
   if (hint == 0 || hint > MI_HINT_MAX) {   // wrap or initialize
     uintptr_t init = MI_HINT_BASE;
     #if (MI_SECURE>=1 || defined(NDEBUG))  // security: randomize start of aligned allocations unless in debug mode
-    const uintptr_t r = _mi_theap_random_next(mi_theap_get_default());
-    init = init + ((MI_HINT_ALIGN * ((r>>17) & 0xFFFFF)) % MI_HINT_AREA);  // (randomly 20 bits)*4MiB == 0 to 4TiB
+    mi_theap_t* const theap = _mi_theap_default();
+    // avoid `mi_theap_get_default()` here, it may lazily initialize the thread
+    // allocator and recurse into arena allocation while `arena_reserve_lock` is
+    // already held, leading to a self-deadlock on the same non-recursive lock
+    if (!mi_theap_is_initialized(theap)) return NULL;  // no hint as we lack randomness at this point
+    const uintptr_t r = _mi_theap_random_next(theap);
+    init = init + ((MI_HINT_ALIGN * ((r>>17) & 0xFFFFF)) % MI_HINT_AREA);  // (randomly 20 bits)*4MiB == 0 to 4TiB
     #endif
     uintptr_t expected = hint + size;
     mi_atomic_cas_strong_acq_rel(&aligned_base, &expected, init);

[chirizxc]

Oh, it looks like the bug isn't reproducible in the dev3 branch.