fix(reach): integrate existing bounded queue-admission fix for cold builds #557

Summary

Integrate the existing user-authored queue-admission fix into this fork's cold Reach limiter. This is follow-up work on the limiter from #341 / #358, not a newly discovered defect or a request to start a competing implementation.

Verified baseline: ff1acc02255b5c6ab852e4cf931f85dbe3043ef9.

Current behavior

cmd/server/node_reach_cold_limit.go limits active cold builds to two and bounds each wait to three seconds, but does not cap the number of distinct-key build leaders waiting for a slot. Once the active slots are occupied, acquire allocates a timer for every additional leader. A time limit is not a queue-size limit.

The existing frontend retry work in #530 / #547 improves busy feedback but does not bound backend admission. The opt-in observation-hop index proposal in #536 is also a different concern.

Starting point and ownership

Reuse and review the user's existing fix rather than independently reimplementing it. Its source branch currently has an open PR. Record any fork-specific adaptation and retain attribution.

Existing user-authored fix:
commit 1d2cd99ddb550570aa2a7c34e79eee30c9db5f8d
nullrouten0/Baymc-CoreScope PR 1 (open at issue preparation)
Source repository: Kpa-clawbot/CoreScope
Issue number: 2264
Related pull request number: 2249

Bounded scope

• Add a fixed cap of eight queued distinct-key build leaders, in addition to the existing two active builds.
• Reject excess admission immediately using the existing HTTP 429 response and Retry-After: 2, before allocating another wait timer or starting database work.
• Release a queued admission on acquisition, request cancellation and timeout; release it exactly once.
• Preserve warm-cache bypass, the active-build limit, the bounded wait and same-key singleflight coalescing.
• Keep the limiter inside the cold-build leader path after the cache recheck. Do not make cache hits or every same-key follower consume queue admissions.
• Do not change database schema, publication/deployment behavior or Reach result semantics.

This does not claim to bound all HTTP requests or singleflight followers. A total request/follower limit would be a separately scoped decision. The cap can initially follow the existing limiter's fixed-default pattern; track any later operator/customizer exposure separately rather than expanding this fix.

Acceptance and regression tests

• Adapt the existing fix only after checking whether that branch or another fork PR has already progressed.
• Occupy two active slots and admit eight queued leaders; the next distinct leader returns 429 promptly without another timer or DB query.
• Prove queue admissions can be reused after each exit path: acquire, timeout and cancellation, including cancellation/acquisition races under -race.
• Assert active builds never exceed two and queue occupancy never exceeds eight.
• Cached requests bypass the saturated limiter; identical-key requests still share one build.
• Queued or rejected work does not increase database-pool activity; only an admitted active build starts its query.
• Add a mutant that removes admission accounting/cap and show the overflow regression fails, then passes with the fix.
• Run focused server tests, -race, go vet and the relevant server suite in an isolated local environment. Do not use staging or production for the regression.

Evidence boundary

This issue is based on source inspection of the stated baseline and the existing fix provenance. No new load benchmark, live-host test or runtime reproduction was performed while preparing it.