[BUG] Flaky test: test_semaphore_caps_concurrent_callers_at_exactly_cap (peak>num_callers under concurrent load)

[jsbattig]

Summary

tests/unit/global_repos/test_verification_semaphore.py::TestSemaphoreConcurrencyCap::test_semaphore_caps_concurrent_callers_at_exactly_cap is flaky: it fails intermittently under concurrent CPU load, with an internally impossible observed value, but passes deterministically in isolation.

Evidence

• Failed once during the v11.28.0 full regression run, while the machine was saturated (fast-automation took 13:44 vs. the normal <10 min — a loaded-machine signature). Failure state captured in the traceback:
  shared = _SharedState(active=0, peak=7, released=True, ...)
  with cap=2, num_callers=3. A peak=7 is physically impossible for a test that starts only 3 caller threads — a correct-or-broken semaphore would yield peak <= num_callers = 3. A value above the caller count can only come from the _SharedState/timing coordination in the test harness misbehaving under CPU saturation (counter contamination / lost condition-variable wakeups), not from a real semaphore-cap violation in production code.
• Passed 5/5 in isolation (idle machine, immediately after the regression completed).
• Passed 12332/0 twice earlier the same day in isolated fast-automation runs (the [BUG] Temporal reuse-seam passes collection name (not model name) to tokenizer loader -> 401 HuggingFace per query, dead up-front-embed optimization, fails e2e Phase-3 log gate #1321 engineer's own gate and the earlier full run), both with the v11.28.0 fixes already present.

Classification

Pre-existing flaky test (test-harness concurrency race), NOT a product bug and NOT a v11.28.0 regression. verification_semaphore is entirely unrelated to the v11.28.0 changes (#1321 temporal-query provider resolution, #1322 langfuse test isolation).

Impact

Intermittent false failures in fast-automation.sh when the suite runs under heavy concurrent load. No production impact.

Fix Direction

Harden the test harness against load-induced flakiness: make _SharedState.peak/active bookkeeping and the "wait until cap active, then release" coordination robust to scheduler delays (e.g. bounded condition-variable waits with explicit re-checks, or assert on the semaphore's own accounting rather than a wall-clock-timed thread snapshot). The peak > num_callers observation is the key clue — the peak counter is being incremented/read outside proper synchronization. Reproduce by running the test under artificial CPU load (e.g. concurrent stress/parallel pytest) rather than in isolation.

Severity

priority-4. Test-infrastructure flake; false-positive noise only.