Multithreaded WASM build: -sPTHREAD_POOL_SIZE too low for multi-engine apps

[kbatchu]

Problem

The multithreaded WASM build (@ladybugdb/wasm-core/multithreaded) exhausts the Emscripten thread pool when running alongside other WASM engines (DuckDB-WASM, WebLLM) in the same browser tab.

Console output:

[LadybugDB] OPFS mounted at /persist
Tried to spawn a new thread, but the thread pool is exhausted.
This might result in a deadlock unless some threads eventually exit or the code explicitly breaks out to the event loop.
If you want to increase the pool size, use setting `-sPTHREAD_POOL_SIZE=...`.

Context

Our application runs three WASM engines concurrently in one browser tab:

• DuckDB-WASM (geospatial SQL queries)
• WebLLM (in-browser LLM inference via WebGPU)
• LadybugDB-WASM (graph database)

OPFS persistence (PR #360) works correctly — the mount succeeds. But when LadybugDB tries to spawn threads for database operations, the browser's pthread pool is already partially consumed by the other engines, causing exhaustion and deadlock.

Reproduction

1. Load DuckDB-WASM + WebLLM in the same page
2. Import @ladybugdb/wasm-core/multithreaded
3. Mount OPFS: await fs.mountOpfs('/persist')
4. Create a database: new lbug.Database('/persist/db') — thread pool exhaustion

Suggested Fix

Increase -sPTHREAD_POOL_SIZE in the multithreaded CMake build configuration (currently defaults to Emscripten's default of 4). A value of 8-12 would accommodate multi-engine browser applications.

Alternatively, consider making it configurable at runtime or using -sPTHREAD_POOL_SIZE_STRICT=0 to allow dynamic growth.

Environment

• @ladybugdb/wasm-core v0.15.4-dev.1 (April 3 nightly, includes PR fix: gate -lopfs.js behind pthreads to fix OPFS crash #360)
• Chrome 135 on Windows 11
• COOP/COEP headers enabled
• SharedArrayBuffer available

Related

• PR fix: gate -lopfs.js behind pthreads to fix OPFS crash #360 (OPFS fix) — merged, working
• Discussion OPFS support for LadybugDB Wasm ? #305 (OPFS persistence tracking)

[kbatchu]

Update after testing: The issue is not that the pool size is too small — it's that threads are eagerly pre-spawned at module init (while (pthreadPoolSize--) { allocateUnusedWorker(); }).

When LadybugDB WASM loads alongside DuckDB-WASM and WebLLM (which already have 10+ workers running), pre-spawning 8 or even 16 additional workers exceeds the browser's per-origin worker limit.

Workaround we're testing: Patching pthreadPoolSize from 8 to 0 in the multithreaded lbug_wasm_worker.js. This disables eager pre-spawning — threads are created on-demand only when the database needs them. This should work for single-connection use cases (our scenario).

Suggested fix: Either:

1. Set -sPTHREAD_POOL_SIZE=0 in the multithreaded CMake build (on-demand threads)
2. Or expose a runtime config so applications can control pre-spawn count before init

Currently testing with pool size 0. Will update with results.

[kbatchu]

Result: pthreadPoolSize=0 also fails. The issue is not pre-spawning — it's that the browser's web worker limit is exhausted by DuckDB-WASM + WebLLM before LadybugDB even tries to spawn its first thread.

With pthreadPoolSize = 0, the first on-demand spawnThread call still fails because there are no available worker slots.

Root cause: The multithreaded WASM build requires pthreads, which requires web workers. In a multi-engine browser app (DuckDB-WASM + WebLLM + LadybugDB), there aren't enough workers available for all three.

What would actually fix this:

1. Single-threaded OPFS support — WasmFS OPFS backend without requiring pthreads (may not be possible with current Emscripten architecture)
2. A shared worker pool across WASM modules (browser-level change)
3. Sequential init — defer LadybugDB init until after DuckDB/WebLLM release their init workers (we'll investigate this)

Reverted to v0.15.3-dev.1 (single-threaded + IDBFS) for production stability.

[kbatchu]

Testing a new approach: sequential initialization.

Instead of starting all WASM engines concurrently, we now init LadybugDB FIRST (before DuckDB-WASM + WebLLM), let it mount OPFS and open the database, then start the other engines after LadybugDB's init workers are released.

Also patched pthreadPoolSize from 8 to 4 to reduce the pre-spawn footprint.

Will update with results. The core request remains valid — the default pool size of 8 is too aggressive for multi-engine browser apps and should be configurable or reduced.

[kbatchu]

Root cause identified: The multithreaded build spawns two separate worker pools:

1. Emscripten pthreads (pthreadPoolSize in the worker JS) — 8 workers by default
2. threads.js WorkerPool (navigator.hardwareConcurrency) — 16 workers on a 16-core machine

Combined: 24 web workers spawned just for LadybugDB. Browsers limit web workers to ~20 per origin. The pool is exhausted before any database operation runs.

This happens in complete isolation — no other WASM engines needed to reproduce. The threads.js dispatcher should use a much smaller pool (1-2 workers) since LadybugDB typically has a single connection.

Tested with sequential init (LadybugDB first, before DuckDB/WebLLM) and with pthreadPoolSize patched to 4 and 0 — all fail because the threads.js pool alone exceeds the limit.

Staying on v0.15.3-dev.1 (single-threaded + IDBFS).

[adsharma]

pthreadPoolSize=0 and dynamic scaling is the right solution here. A static build parameter in ladybug cmake can't size the threadpool for an arbitrary application using multiple databases.