Survive a refused thread creation in the GC helper pool, the collector thread, and the JIT worklist

[robobun]

Problem

• A process whose own children fill its pid budget (systemd-run -p TasksMax=64 bun spawn80.js, or prlimit --nproc=64) aborts at its next full GC: ASSERTION FAILED: success in Thread::create (Threading.cpp:330), reached from AutomaticThread::start via ParallelHelperPool::didMakeWorkAvailable in Heap::runBeginPhase. Node under the same limit reports EAGAIN for the spawns that do not fit and keeps running.
• The heap helper threads, the collector thread, and the JIT compiler threads are AutomaticThreads. They exit after 10 seconds of idle time and are created again on demand, so every GC and every tier-up after the limit is reached is a pthread_create that can fail. Thread::create has no failure path.

Fix

• Thread::tryCreate returns nullptr when establishHandle fails. Thread::create now wraps it and keeps the crash.
• AutomaticThreadCondition::create(StartFailure): with Retry, a failed AutomaticThread::start leaves the thread without an underlying thread and notifyOne/notifyAll return false. The default Crash keeps the old behavior for every other user.
• ParallelHelperPool uses Retry: the client drains the work itself. JSC::Heap uses Retry: the mutator takes the conn back and, when it waits for the collection, runs the collector's phases too (see collectAsTheCollectorBecauseCollectorThreadCouldNotStart). JITWorklist uses Retry: wakeThreads counts only threads that exist, and waitUntilAllPlansForVMAreReady cancels plans nobody can compile instead of waiting forever.
• Verified in Bun (debug, ASAN) by relinking its prebuilt WTF and JSC archives with the patched objects. Under prlimit --nproc=64 as an unprivileged user: the spawn repro prints { ok: 58, err: 22 } and exits 0 (before: abort). A 300k-object heap with sync and async GCs while the pid budget stays full completes. A hot loop that needs a compiler thread that cannot be created, followed by deleteAllCode, returns in 48 ms and the JIT recovers once threads are available again. With no limit, 20 sync GCs take the same time as before (1150 ms vs 1152 ms), and a sample of Bun's suites has the same result set.

Background

• The "conn" is the right to run GC phases. Either the mutator (at stopIfNecessary polls) or the collector thread holds it. requestCollection usually steals it for the mutator; relinquishConn hands it to the collector and that is where the collector thread gets created.
• Phases that need a stopped world run under the conductor that has the conn. The concurrent phase is the collector's: with StochasticSpaceTimeMutatorScheduler it ends when marking terminates or when the mutator allocates its headroom. A mutator parked in waitForCollector allocates nothing, so without a collector that phase never ends. The patch has the waiting mutator clear its conn bit and call collectInCollectorThread; stopTheMutator hands the conn back as soon as the world must stop, because the mutator has heap access.
• JITWorklist::m_numberOfActiveThreads is incremented per notify and decremented when a worklist thread finds no work. A notify that started no thread used to leave it too high.

Notes

Fuzz ledger repro (Bun 1.4.0):

// systemd-run --scope -p TasksMax=64 bash -c 'exec bun spawn80.js'
const { spawn } = require("child_process");
let ok = 0, err = 0;
for (let i = 0; i < 80; i++) { const p = spawn("/bin/sleep", ["2"]); p.on("spawn", () => ok++); p.on("error", () => err++); }
setTimeout(() => console.log({ ok, err }), 1000);

Without a cgroup, prlimit --nproc=64 setpriv --reuid=nobody bun spawn80.js reproduces it.

Iterations of this patch and what they showed:

• Helper pool only: the GC then reached finishRelinquishingConn and aborted on the collector thread instead.
• Collector Retry with the mutator keeping the conn and spinning: relinquishConn loops while the conn bit is set, so it retried pthread_create on every iteration (130k attempts in 10 s). finishRelinquishingConn now reports the retake and relinquishConn stops.
• Mutator keeps the conn and sleeps until m_scheduler->timeToStop(): with the stochastic scheduler that is infinity while the mutator allocates nothing, so a sync GC never finished once the pid budget stayed full. Hence the collector stint on the mutator thread.

The debug-only LOG_ERROR in establishHandle still prints one line per refused thread. Release builds print nothing.

Not changed: WorkerPool, Wasm::Worklist, SequesteredAutomaticThread keep StartFailure::Crash. Every direct Thread::create caller keeps the crash.