fix(tests): wait for provider embedding batch completion before assertions

[seonghobae]

What

Fixes a race condition in two tests in tests/test_provider_embedding_batch_backend.py:
test_unknown_tokenizer_uses_authoritative_provider_usage and
test_unknown_tokenizer_byte_bound_never_becomes_recorded_usage.

What surfaced it

PR #992 ("docs(gap): record isolated application service boundary" — a docs-only PR that does
not touch any embedding/token-counting code) failed its "Full unit and contract suite" check on
head 0e483013ced2705a027576717c5dd5e0fd1609e9 (base main@c594b682) with exactly one failure:

FAILED tests/test_provider_embedding_batch_backend.py::test_unknown_tokenizer_byte_bound_never_becomes_recorded_usage[한글🙂é] - KeyError: 'total_tokens'
1 failed, 3389 passed, 2 skipped in 710.42s

Only one of the three parametrized inputs was reported as failed, which looked at first like a
Unicode-specific bug in the "unavailable tokenizer byte-bound" fallback path.

Root cause

It is not content-dependent — it is a genuine race condition, unrelated to which parametrize
case runs.

Both tests build a CostRoutingCoordinator with a non-mock (https://...) embedding agent.
CostRoutingCoordinator.__init__'s remote_embedding_agents branch selects the async
ProviderEmbeddingBatchBackend, whose _run_job executes the runner in a background
ThreadPoolExecutor thread rather than inline.

complete_embeddings_batch()'s own docstring says so explicitly: "Local backends complete
immediately. Callers that require a synchronous provider result pass wait_timeout; a timed-out
queued job is cancelled so the synchronous surface does not leave orphaned work." Both tests
called complete_embeddings_batch(...) with no wait_timeout, so nothing blocked the calling
thread until the background job actually finished.

Most runs "win" the race because the synthetic runner returns near-instantly, but under CI load
(a 710s, 3389-test run) the calling thread can reach embeddings_batch_document() while the job
is still "running". CostRoutingCoordinator._embeddings_batch_document_locked's
not-is_complete early return produces a document with only batch_id/status/backend/
model/embeddings — total_tokens (and cost_micro_usd, etc.) is simply absent — which is
exactly the reported KeyError: 'total_tokens'.

Every other provider-backed complete_embeddings_batch() call in the same test file already
passes wait_timeout for this reason (e.g.
test_runtime_added_remote_embedding_member_uses_provider_backend,
test_provider_embedding_requests_are_sharded_by_the_existing_token_limit), and
test_unknown_tokenizer_rejects_missing_provider_usage explicitly polls instead of relying on
synchronous completion — these two tests were the only outliers.

Verification

• Deterministically reproduced the race with a controllable blocking runner: without
  wait_timeout, complete_embeddings_batch() returns a document missing total_tokens while
  the job is still "running"; with wait_timeout=1, the document is "completed" with the
  correct total_tokens. Confirmed the input text's specific Unicode content is irrelevant to
  the mechanism.
• Confirmed the failure is present identically on unmodified main (git stash before/after) —
  not introduced or masked by this diff, and unrelated to PR docs(gap): record isolated application service boundary #992's own (docs-only) changes.
• tests/test_provider_embedding_batch_backend.py (all 20 tests) — passing, repeated 5x.
• pytest -k "embedding or token_counting or cost_router" — 261 passed, 1 skipped.
• Full suite (python -m pytest -q, Python 3.12) — 3388 passed, 2 skipped, 2 failed; both
  failures (test_psychometric_routing.py::test_fast_mlsirm_fit_uses_judge_acceptance_item_for_context_score,
  test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable) are
  pre-existing local-sandbox artifacts unrelated to this change (this sandbox's git proxy could
  not fetch the fast-mlsirm Rust wheel through the normal uv run --locked path, so the venv
  used here was hand-assembled and is missing native pieces real CI has); both reproduce
  identically on vanilla main with this diff stashed out, so they are not caused by this PR.

No production code changed.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

---

Generated by Claude Code

Summary by CodeRabbit

• 테스트
  • provider 임베딩 배치 작업이 백그라운드에서 완료될 때까지 동기 호출이 대기하는 동작을 검증하도록 테스트를 업데이트했습니다.
  • 파라미터화된 테스트에 비동기 처리 및 대기 동작에 대한 설명을 추가했습니다.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 0eb1bae2-d49c-409d-b38d-d3ccf24e6f62

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 799fec5b-a37b-4dbe-98f6-fec82180660a

📥 Commits

Reviewing files that changed from the base of the PR and between c594b68 and 7c0d420.

📒 Files selected for processing (1)

• tests/test_provider_embedding_batch_backend.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

Provider 임베딩 배치 테스트가 백그라운드 작업 완료까지 대기하도록 수정되었습니다. 관련 테스트에는 비동기 완료와 문서 기록 사이의 스케줄링 경합을 설명하는 주석이 추가되었습니다.

Changes

Provider 임베딩 배치 테스트

Layer / File(s)	Summary
배치 완료 대기 검증 tests/test_provider_embedding_batch_backend.py	두 테스트의 complete_embeddings_batch() 호출에 wait_timeout=1을 추가했습니다. 파라미터화된 테스트에는 비동기 작업이 완료되기 전에 문서가 반환될 수 있는 스케줄링 경합을 설명하는 주석을 추가했습니다.

Estimated code review effort: 1 (매우 간단) | ~5분

Merge Risk: ⚪ Minimal · up to 7c0d4

Provider embedding batch tests now wait for asynchronous work to complete before checking recorded token usage, eliminating the timing-related test failure without changing production behavior.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	PR 제목은 두 provider embedding batch 테스트에서 완료를 기다리도록 수정한 핵심 변경을 정확하고 구체적으로 설명합니다. 비동기 작업의 race condition 수정이라는 목적도 명확합니다.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/embedding-batch-wait-timeout-race

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Hourly loop check-in — 2026-09-04

Progress toward green: none since creation (7c0d420 at 2026-09-03T19:55:47Z). All required checks (Full unit and contract suite, NIM benchmark coverage..., Security, Security Scan, SAST Semgrep, Fuzz, OSV-Scanner PR, CodeQL PR, Scorecard PR, noema-review, etc.) are still queued with no runner assigned, 8+ hours later. This is the same severe org-wide Actions capacity congestion tracked elsewhere (.github#712), not a defect in this PR. No re-run was triggered since nothing has actually failed — only queued jobs, which should not be re-triggered.

Duplicate-effort note: PR #1025 (fix/embedding-batch-race-wait-timeout, opened 2026-09-02) independently fixes the exact same root cause with the same wait_timeout=1 change to the same two tests in tests/test_provider_embedding_batch_backend.py. #1025 has already been reconciled onto a fresher main twice (most recently merge commit b2c58154 on 2026-09-03, which also picked up the unrelated test_admin_contract.py import json fix from #1035) but has since fallen behind main again. Both PRs are correct and minimal; no code defect in either. Flagging so review/merge doesn't land both — whichever gets required-check evidence first should merge, and the other can then be closed as superseded (not by me — only a human/the merge scheduler should do that per this repo's merge governance). No action taken on either PR's code.

---

Generated by Claude Code

[seonghobae]

Adjudication evidence (host 1 session, 2026-09-06 KST; full report with commands in #1080). Nothing here closes, flips, or retargets anything — the decision is the opener's.

Near-duplicate of #1025 (same wait_timeout=1 edits; differs only by your comment block at :127-137 and #1025's CHANGELOG.d fragment). Neither direction was verified as full containment, so neither is closable yet; folding #1025's fragment in here would make this the survivor. Also: no CHANGELOG.d fragment of its own, and the body's full-suite evidence comes from a hand-assembled sandbox venv (two unrelated failures), not exact-head hosted evidence. #1002 and #1017 touch the same test file; #1053 changes complete_embeddings_batch wait semantics (+68) — land this before those.

[seonghobae]

CI note and push: head 95bc9a7e7 → 5e2e5f16 (a base merge only; no change to the PR's own commits; 14 h since the previous push).

The two red checks on the previous head were org-wide classes, not this diff's: opencode-review (run 33865732266) failed in 7 s at "Fail closed without a current-head OpenCode verdict" — the required job's designed wait for a dispatch handler that the .github#1929 actor-gate mismatch rejects — and noema-review (run 33865732290) failed after 96 s with HTTP Error 502: Bad Gateway; caller attempts=1, the pre-pin-bump gateway stall that .github efb892692 (co#1081, merged 2026-09-06T03:01Z) removed. That fix reaches this PR through the central workflow, which binds its sidecar at run creation, so it needed a fresh run: this merge of main@414f2297 creates one on the new pin and the merged preflight launcher (.github#1947/#1949).

Local gate on the merged tree (python -m pytest tests -q): 3392 passed, 2 skipped, 3 failed — all three reproduce identically on origin/main in this environment and none is in code this PR touches (test_psychometric_routing needs fast_mlsirm, test_telemetry needs the OTLP HTTP exporter, and test_spend_analytics::test_exact_output_without_prompt_usage_is_explicitly_unavailable asserts mixed against tokenizer on main too); git diff --check clean. The PR's own embedding-batch tests pass.

---

Generated by Claude Code

[seonghobae]

CI note on head 5e2e5f16: both CodeQL compatibility analysis shards (python, javascript-typescript; run 34018736499) failed in 5–6 s with "CodeQL scan dispatched. The dispatch workflow will rerun this exact failed CodeQL job after publishing its terminal verdict." That is the compat job's designed fail-closed wait: it dispatches a codeql-scan handler to the .github hub and exits red so it releases its runner, expecting the handler to re-run it. The handler is rejected before it can run — .github's codeql-scan-dispatch.yml authorizes the dispatch only when actor and sender both equal vars.OPENCODE_REPOSITORY_DISPATCH_ACTOR, which still names github-actions[bot] while the dispatch arrives as opencode-agent[bot] (.github#1929; 231 such handlers rejected in three hours this morning) — so these two shards cannot turn green by anything done on this branch. Not this PR's; no re-run (a re-run repeats the same dispatch); nothing to push. The owner-side remedy is the variable value.

---

Generated by Claude Code

[seonghobae]

CI note on head 5e2e5f16, the two remaining red checks — and the first is not a gateway failure, which makes it different from every other review failure I have triaged today.

noema-review (run 34018735728, job 101451677626, 32 min) reached the model and got a verdict. It failed in "Prepare Noema model verdict" with Noema model output failed local validation: Noema approve cannot contain a confirmed adversarial probe; caller attempts=1, duration=1294.6s, phase=validating, served_model=deepseek-ai/deepseek-v4-flash-0731. phase=validating, not response_error: the gateway served a completion and Noema's own local gate rejected its content — the model returned an approval whose body carried a confirmed adversarial-probe marker, which the gate refuses by design. So the review pipeline worked end to end here; the model's answer was the problem.

Supporting evidence from artifact 9986228615, and it is the healthiest preflight of the day: ready 6 / probed 16 / skipped 4 / deferred 2 — both keys' flash and pro plus both keys' llama-3.2-11b, which the pre-#1949 fill never reached, with the two rate-limited OpenRouter routes kept as deferred failover. That is #1947 and #1949 working exactly as projected. In serving, 15 timeouts of which 9 were followed by a circuit_failure record — the opposite of the Strix runs I measured this morning (176 timeouts, 0 records), which confirms the split: Noema's no-tools path goes through _invoke → classify_provider_transport_failure and does record, while the tool-bearing passthrough path co#1082 fixes does not.

opencode-review (run 34018735705, job 101455702904) failed in 7 s at "Fail closed without a current-head OpenCode verdict" — signature 1's designed wait, blocked by the .github#1929 actor-gate variable, as already noted here for the CodeQL shards.

Neither is this diff's, so nothing to push. The Noema verdict is a content rejection on a specific model output, not a defect in the embedding-batch change; a fresh run may well produce a valid verdict from the same pool, but this PR's re-run allowance stays unspent until the queue is shallower — the run took 32 minutes and the pool is now rate-limited again.

---

Generated by Claude Code