fix(bridge): cap the retry path by CODER_AGENT_SLOTS + share coder load across the tick - #344
Conversation
…ad across the tick reconcile_failures (the Failed-Workload retry recreation) was the third workload-creation path and the only one still uncapped: #340 capped the issue-claim loop and the pr-fix drain, but a retried coder was recreated regardless of coder capacity. Observed in-cluster: wl-...-616 retried (attempt 3/3) onto the local coder while a pr-fix coder was already running -- two coders on the single-slot 27b. - reconcile_failures now resolves the coder before the delete, checks free_slots, and defers a retry whose coder is at capacity (leaves the Failed tombstone; list_failed re-offers it next tick). - coder_load is computed once before the retry pass and threaded through all three passes (reconcile -> issue-claim -> pr-fix drain). reconcile mutates it in place, so a retried coder is visible to the later passes: the three paths draw from one pool instead of each independently filling the same slot.
There was a problem hiding this comment.
AI Automated Review
Full PR review.
Analysis engine: MiniMax-M3@https://litellm.jory.dev/v1 (anthropic) — escalated (fast_low_confidence)
Recommendation
Approve. The change is a focused, well-scoped follow-up to PR 340 that closes the remaining retry-path leak by capping reconcile_failures against CODER_AGENT_SLOTS and threading a single shared coder_load dict through the three workload-creation passes in one tick. CI is green (test + docker both success) and the new tests cover the load-bearing cases (defer-at-capacity, in-place slot draw, uncapped-when-empty).
Change-by-change findings
bridge/main.py
- Moves
coder_load = load_by_coder_agent() if cfg.coder_slots else {}up to the top of the tick, before the retry pass. This is the right place: it is computed once and shared across the retry, issue-claim, and pr-fix passes. PR PR 340 had two independent copies; PR 344 correctly collapses them onto one. - Threads
agent_load=coder_load, agent_slots=cfg.coder_slotsintoreconcile_failures. The signature change is additive (both haveOptionaldefaults), so existing call sites outside this tick keep working. - Removes the now-redundant second
coder_loadassignment further down (the line previously assigned inside the issue-claim / pr-fix region). Verified by reading the diff: only oneload_by_coder_agent()call remains in the tick.
bridge/retry.py
- Adds
agent_load: Optional[dict] = Noneandagent_slots: Optional[dict] = Nonekeyword args toreconcile_failures. Backward-compatible. - Resolves
coder_agentbeforedelete_workload. This is correct: deleting then failing the capacity check would leave the workload gone and the tombstone lost, which would be a regression versus the documented "leave the Failed tombstone solist_failed()re-offers it" contract. Doing the resolution up-front also avoids re-runningcoder_agent_for(...)later (the old call inside the try-block is gone). - The defer path emits
f"{name}:retry-deferred:coder-busy:{coder_agent}"andcontinues without deleting or recreating, exactly matching the contract described in the PR body and the new test. - After a successful recreate, the function mutates the shared dict in place (
load[coder_agent] = load.get(coder_agent, 0) + 1). Becauseloadis the same object the caller passed in, downstream issue-claim and pr-fix passes in the same tick will see this draw — which is the whole point of the "one pool" refactor. - The
free_slots(coder_agent, load, slots) <= 0guard is gated byif slots, so the legacy "noCODER_AGENT_SLOTSconfigured" path is unchanged. The newtest_reconcile_retry_uncapped_when_no_slots_configuredtest pins that behavior.
tests/test_retry.py
test_reconcile_retry_deferred_when_coder_at_capacity— exercises the defer path withagent_load={"coder": 1}, agent_slots={"coder": 1}, asserts the result string and that nothing was deleted or created. Matches the production scenario the PR body describes.test_reconcile_retry_proceeds_and_draws_shared_load_when_slot_free— passesload={}intoreconcile_failuresand asserts the function drew it down in place to{"coder": 1}. This is the load-bearing assertion: it verifies the shared-mutation contract without which the issue-claim / pr-fix passes would double-count and re-fill the same slot.test_reconcile_retry_uncapped_when_no_slots_configured—agent_slots={}with a non-emptyagent_loadstill recreates, confirming theif slots:guard.
Cross-cutting checks
- No auth / public-route / secret / migration risk. Classification
risk_flags: []agrees; nothing in the diff touches env parsing, HTTP routes, RBAC, or schema. The new keyword args are typedOptionaland default toNone, so callers that pre-date PR 344 remain source-compatible. - Failure semantics preserved. The original code wrapped the delete + recreate in one
try/exceptthat emittedf"{name}:retry-error:{e}". The refactor splits that: defer is a cleancontinue(no exception path), and thetry/exceptstill wrapsdelete_workload,build_workload, andcreate_workload. An infra error during delete or create still surfaces as:retry-error:..., so a wedged delete / 409 race / HTTP failure on the recreate path still continues to the next workload instead of crashing the tick (the comment in the diff explicitly preserves this property from the existing code). - Determinism of result strings. The PR adds a new
:retry-deferred:coder-busy:{coder_agent}result string. No consumer grep hits forretry-deferredexist in the repo today, so this is purely additive — no existing matcher or test will break. (Verified by repo impact scan.) free_slotsimport. Added to the existing import block frombridge.workload. The function exists in that module per the related-code leads and the new tests' use of it (implicitly, via the guard firing).
Standards Compliance
The repository's AGENTS.md is about filing issues for the autonomous loop — reviewer-side ask quoting and named-file scope-overlap rails. This PR's metadata does not include a linked issue body in the corpus, so the issue-quoting rail is not directly verifiable here. The PR title, body, and diff are internally consistent with the linked-history context (PR PR 340 / commit a136630 capped the fresh-dispatch paths; PR 344 explicitly closes the remaining retry-path leak). The diff touches exactly the files one would expect (bridge/retry.py, bridge/main.py, tests/test_retry.py) and does not introduce scope drift into unrelated modules.
Unknowns / Needs Verification
- The exact signature/body of
free_slots(coder_agent, load, slots)inbridge/workload.pywas not read directly (no native tool calls succeeded in this review). The diff's use (free_slots(coder_agent, load, slots) <= 0) is consistent with the existing usage inclaim_one(the related-code context showsfree_slotsalready imported and used inbridge/main.py), and the new test asserts the observable outcome (defer / draw / uncapped). Iffree_slotshad a different arity or naming, the test would not pass — and CI is green — so the function contract is consistent with the diff's usage. - No evidence provider / tool-harness output beyond CI status was generated in this review (the harness issued no tool calls). The CI run for the head commit reports
test: successanddocker: success, which corroborates the PR body's "Full suite: 649 passed."
…nbounded coder concurrency) Three coder-creating paths were uncapped, so a batch of work could recreate coders concurrently and thrash the single-slot local 27B (observed: 6 at once, pipeline to a crawl). #344 capped the issue-claim loop, reconcile_failures, and drain_pr_fixes; this completes the set so NO path can spawn a coder past the cap: - reconcile_pr_fixes same-tier retry (retry / retry-progress): gate the recreate on the current coder's free_slots; defer (leave the tombstone) when full. - reconcile_pr_fixes escalation: gate the next-tier recreate on THAT tier's free_slots (coder-frontier has its own cap), so a burst of escalations cannot exceed the stronger coder either. - redrive_infra: gate the infra-recovery recreate on free_slots; return False to defer (reconcile_infra_parked keeps the marker and retries next tick). coder_load is threaded from run_tick through reconcile_failures -> infra redrive -> reconcile_pr_fixes -> drain, mutated in place, so all paths draw from one shared per-tick pool. Every defer is transient (retry next tick), never a silent BLOCK. Escalation to frontier is still reached -- gated by frontier's own slots, not the local coder's.
…nbounded coder concurrency) (#346) Three coder-creating paths were uncapped, so a batch of work could recreate coders concurrently and thrash the single-slot local 27B (observed: 6 at once, pipeline to a crawl). #344 capped the issue-claim loop, reconcile_failures, and drain_pr_fixes; this completes the set so NO path can spawn a coder past the cap: - reconcile_pr_fixes same-tier retry (retry / retry-progress): gate the recreate on the current coder's free_slots; defer (leave the tombstone) when full. - reconcile_pr_fixes escalation: gate the next-tier recreate on THAT tier's free_slots (coder-frontier has its own cap), so a burst of escalations cannot exceed the stronger coder either. - redrive_infra: gate the infra-recovery recreate on free_slots; return False to defer (reconcile_infra_parked keeps the marker and retries next tick). coder_load is threaded from run_tick through reconcile_failures -> infra redrive -> reconcile_pr_fixes -> drain, mutated in place, so all paths draw from one shared per-tick pool. Every defer is transient (retry next tick), never a silent BLOCK. Escalation to frontier is still reached -- gated by frontier's own slots, not the local coder's.
What
Cap the retry path (
reconcile_failures) byCODER_AGENT_SLOTS, and thread one sharedcoder_loadthrough all three workload-creation paths in a tick.Why
#340 capped the two fresh-dispatch paths (issue-claim loop, pr-fix drain) and made the load counter see pr-fix workloads. But there's a third path —
reconcile_failuresrecreating Failed workloads — and it had no capacity check. Observed in-cluster after deploying #340:wl-616was retried onto the localcoder(qwen 27b) while a pr-fix coder was already running → two coders on the single-slot model. The 3-way drain pileup was fixed; this was the remaining leak.Fix
reconcile_failuresresolves the coder before deleting, checksfree_slots, and defers a retry whose coder is at capacity —…:retry-deferred:coder-busy:<agent>— leaving the Failed tombstone solist_failed()re-offers it once a slot frees.run_tickcomputescoder_loadonce before the retry pass and passes it to all three consumers.reconcile_failuresmutates it in place, so a just-recreated retry is visible to the issue-claim loop and pr-fix drain that run after it — the three paths draw from one pool rather than each independently filling the same slot.Tests
Three added to
test_retry.py(deferred-at-capacity is load-bearing — verified it fails without the guard): defer when the coder is full, proceed + draw the slot down in place when free, and stay uncapped when no slots are configured. Full suite: 649 passed.