Revisit preload_app for stanza via the ADR-017 OMP fix (gated on latency histogram)

[mircealungu]

Why

Live stanza runs preload_app=False + 1 worker because preload_app=True deadlocks after gunicorn's fork() — PyTorch spins up OpenMP/MKL thread pools in the master, and fork() copies the memory image but not the threads, so the child blocks on a futex held by threads that no longer exist. Disabled in 2393b7c1 (Jan 2026).

Consequence: workers don't share models (no copy-on-write), so each worker costs ~5.9 GB (measured). That's why the healthcheck-flapping fix (ops#10) could only afford 2 workers, and why "/health stays answerable" is only probabilistic — it relies on not having ≥2 large (~12 s) tokenizations occupying both workers within the ~60–90 s detection window.

The lead

ADR 017 (asr-single-threaded-compute) solved the identical fork-after-OMP deadlock for the ASR service, four months later: set OMP_NUM_THREADS=1 + MKL_NUM_THREADS=1 → no thread pool → fork() is safe → preload_app=True + CoW model sharing + workers=2. The ADR explicitly says it applies to "any future ML-on-CPU service." Stanza never got this fix — it has no OMP/MKL env vars (only asr does), because the stanza disable predates the ADR-017 discovery.

If it works for stanza, extra workers become nearly free (shared models) → run 5–6 workers → /health answerability becomes robust even under a spike, removing the fragile assumption.

The catch (why it is not a blind flip)

ADR 017 notes single-threaded OMP makes each op 2–4× slower. Invisible for ASR's 3 s clips (sub-second), but stanza's large-text tail is already ~12 s → single-threaded could push it to ~24–48 s, inflating the very tail the healthcheck timeout was just tuned around. The other ADR-017 option (threaded gunicorn, workers=1 threads=N) is blocked because the stanza pipeline is marked not thread-safe.

Plan (data-gated)

1. Let the tokenize-latency histogram (stanza_service: expose tokenize latency as a Prometheus histogram #663) collect a few weeks of baseline p50/p90/p99.
2. In a staging stanza: OMP_NUM_THREADS=1 + MKL_NUM_THREADS=1, preload_app=True, workers>=2; compare the histogram — does the tail inflate unacceptably?
3. Tail acceptable → adopt preload + more workers (cheap), make /health robust, revisit the timeout.
4. Tail too inflated → keep preload off; pursue decoupling /health from the worker pool (progress-based liveness) instead, which sidesteps worker count entirely.

Refs

• ADR 017 (asr-single-threaded-compute)
• Disable commit 2393b7c1
• Latency histogram PR stanza_service: expose tokenize latency as a Prometheus histogram #663
• Healthcheck-flapping fix: mircealungu/zeeguu-ops#10, stanza_service: loosen healthcheck so busy workers aren't killed #662