feat(subagent): rate-limit-adaptive launch scheduling - #6055
Conversation
Four additions to the deny-prefix matcher that let a ruleset express the vectors real exfiltration and sabotage commands use: 1. **cmd.exe single-letter slash flags** (`/f`, `/s`, `/q`) are skippable in any position/order, like `-` flags. Only the single-letter shape skips: multi-character `/`-tokens are real POSIX paths (`/tmp`, `/etc`) and stay positional, so `cp /tmp/new_key ~/.ssh/authorized_keys` cannot hide its operand behind a skipped path. 2. **Mid-rule wildcard**: a rule token of exactly `*` matches zero or more consecutive command tokens regardless of shape, so a rule can anchor on a sensitive tail (`grep * ~/.ssh/id_rsa`, `dd * of=/dev/sda`, `find * -name id_rsa`) without enumerating every flag spelling. The DFS branches are bounded by the existing `seen` set; a trailing `*` degrades to prefix semantics; a leading `*` follows generic wildcard rules (documented, pinned by test). Each wildcard widens the deny face of its rule — rule authors own that discipline. 3. **`.exe` suffix fold on the deny command word**: Windows spells the same binary `cat.exe` or `C:\Windows\System32\cat.exe`; the anchor folds one trailing `.exe` so a `cat ~/.ssh/id_rsa` rule holds. One direction only — a rule naming `.exe` (`control.exe`) keeps requiring it, and `catalog` never matches `cat`. 4. **Rooted absolute-path rules match exactly**: typed File path rules previously only matched after workspace-relative normalization, so a rule pinning an absolute location (a real home, `/root`, a Windows profile, a literal `~` spelling) could never match a call outside the workspace. A rooted-rule-only exact fallback (separators fold to `/`, case folds on case-insensitive platforms, no wildcards) fixes this; relative rules keep their semantics untouched. Also: `ExecPolicyEngine.rulesets` becomes `Arc<RwLock<…>>` so `set_ruleset` through any clone is observed by every clone — a host that clones the engine into long-lived side executors no longer leaves them on a stale ruleset after a live permission update. `approved_for_session` stays deliberately clone-private: a remembered grant is a decision the parent session made for its own calls and must not authorize a delegated call in a cloned executor. Co-authored-by: asto18089 <44870036+asto18089@users.noreply.github.com> Signed-off-by: asto18089 <asto18089@126.com> Signed-off-by: pinvou3-dev <dev@pinvou3.local>
A swarm of sub-agents launched against one shared provider makes parallel 429s a steady state rather than an edge case. This adds rate-limit-adaptive scheduling to the sub-agent launch gate: - `launch_gate` changes from a fixed-capacity `tokio::sync::Semaphore` to a custom `DynamicGate`: capacity is adjustable at runtime (allowed below the active-holder count; surplus holders finish naturally), and permits are handed to waiters pre-counted through a oneshot channel so a cancelled waiter can never swallow a wakeup — the two cancellation paths (stale queue entry skipped by the granter, dispatched permit re-released by Drop) are both covered by tests. A Semaphore cannot do this: the only way to "shrink" one is to swap the Arc, which silently fails while a permit is held. - New `RateLimitGovernor` (owned by the manager, inherited through the `SubAgentRuntime` derive tree via a single spawn chokepoint): a 60s sliding window halves launch capacity at ≥2 rate-limited attempts or >30% ratio (guarded by an attempts≥2 volume floor), pauses new admissions at ≥4 (capacity 0; in-flight children keep running), and recovers additively — one unit per three consecutive successes (AIMD). A pause is lifted only when the window drains; an external launch-concurrency change cannot silently lift it. Queued children probe `recover_if_window_drained` every 5s so the queue cannot freeze when the in-flight fleet finishes before any success arrives. - 429 retries honor `Retry-After`; without it, full-jitter exponential backoff (250ms base, 120s cap) de-synchronizes a fan-out thundering herd. `QuotaExhausted` is not a transient throttle and keeps the existing failure path. The governor only observes and never delays an in-flight call. - `update_runtime_limits` applies launch-concurrency changes to the live gate immediately (previously deferred until the fleet was idle because the old semaphore swap silently failed while children held permits). The queued-acquire future is pinned across probe iterations so a long pause does not leak one stale queue entry per tick per queued child. Co-authored-by: asto18089 <44870036+asto18089@users.noreply.github.com> Signed-off-by: asto18089 <asto18089@126.com> Signed-off-by: pinvou3-dev <dev@pinvou3.local>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| // spawned agent and its whole descendant tree report 429s/successes | ||
| // to the adaptive scheduler. Runtimes built outside a manager (tests, | ||
| // tool-only runtimes) keep `governor: None`. | ||
| runtime.governor = Some(Arc::clone(&self.governor)); |
There was a problem hiding this comment.
🟡 Cross-provider throttles share one governor
Cross-provider children stamp the same governor, so one provider's 429s pause every provider's launches. bind_profile_provider can route siblings to independent providers.
Learn more
The manager owns one governor and stamps it onto every spawned runtime. Fleet profiles can replace a child runtime's client with another configured provider through bind_profile_provider. All those children still report attempts and 429s into the same window, and the shared gate applies the resulting pause to every direct child.
Example: A DeepSeek child produces four 429s while an independently routed OpenRouter child remains healthy. The shared governor sets the launch gate to zero, so new OpenRouter children wait up to the recovery window despite no OpenRouter throttling.
Recommended fix: Key governors and launch admission by resolved provider identity, or restrict adaptive observations to a manager whose children all share one provider. Preserve the configured aggregate launch ceiling when combining provider-specific gates.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if !state.paused { | ||
| self.gate.set_capacity(state.max_capacity); | ||
| } |
There was a problem hiding this comment.
🟡 Runtime updates erase active throttling
During a non-paused throttle, set_max_capacity restores full capacity even when the configured limit is unchanged. Any runtime settings update then releases queued launches while recent 429s remain.
Learn more
The governor represents throttling only through the gate's current capacity. A multiplicative decrease lowers that capacity but leaves paused false. update_runtime_limits calls set_max_capacity for every runtime settings update, so these lines overwrite the reduced capacity with the configured ceiling.
Example: The configured limit is 8 and two 429s reduce the gate to 4. Changing only max_spawn_depth invokes the runtime update with the same launch limit of 8, immediately reopening all eight slots while both 429s remain in the 60-second window.
Recommended fix: Track the adaptive capacity separately from max_capacity. On configuration changes, clamp the adaptive capacity down to a reduced ceiling, but do not raise it except through additive recovery or an explicit policy reset.
Was this helpful? React with 👍 or 👎 to provide feedback.
| pub use approval_mode::ApprovalMode; | ||
|
|
||
| use std::collections::HashSet; | ||
| use std::sync::{Arc, RwLock}; |
There was a problem hiding this comment.
🔍 Execution-policy changes broaden PR scope
The PR also changes deny wildcards, path matching, Windows command handling, and ruleset clone semantics. Split this trust-boundary work for focused review.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Problem
A swarm of sub-agents launched against one shared provider makes parallel 429s a steady state rather than an edge case. The launch gate is a fixed-capacity
Semaphore, whose capacity cannot actually shrink (the only way is swapping theArc, which silently fails while any child holds a permit — whyupdate_runtime_limitstoday only applies launch-concurrency changes when the fleet is idle), and nothing observes provider throttling: every queued child retries in lockstep and the fan-out hammers the same rate limit.Fix
DynamicGatereplaces theSemaphore: runtime-adjustable capacity (allowed below the active-holder count; surplus holders finish naturally, new admissions block untilactive < capacity), FIFO waiters receiving pre-counted permits through a oneshot channel. Cancellation is safe on both paths — a waiter cancelled before the grant leaves a stale queue entry the granter skips, a waiter cancelled after the grant drops the dispatched permit whoseDropre-releases the slot — so a lost wakeup cannot occur (stress-tested with a timed drain assertion).RateLimitGovernor(owned by the manager, stamped onto every runtime at the singlespawn_background_with_assignment_optionschokepoint, inherited through the derive tree): a 60s sliding window drives AIMD — ≥2 rate-limited attempts or >30% ratio (with an attempts≥2 volume floor so a two-call fleet doesn't halve on one blip) halves capacity; ≥4 pauses new admissions (capacity 0; in-flight children unaffected); three consecutive successes add one unit back. A pause lifts only when the window drains; an external launch-concurrency change routes through the governor and cannot silently lift it. Queued children proberecover_if_window_drainedevery 5s, so the queue cannot freeze when the entire in-flight fleet finishes before any success arrives.Retry-Afterhonored when present; otherwise full-jitter exponential backoff (250ms base, 120s cap) de-synchronizes a fan-out that was 429'd by the same response.QuotaExhaustedis deliberately not reported — quota is a billing condition, not a transient throttle. The governor never delays an in-flight call; it only decides whether new launches may be admitted.update_runtime_limitsapplies launch-concurrency changes to the live gate immediately (previously deferred while children ran, for the semaphore-swap reason above).Tests
Governor suite (14 tests on this branch): window statistics, AIMD up/down boundaries, pause/recover/time-heal, gate cancel re-dispatch, queued-cancel branch, and a multi-task abort/pause stress test that must drain back to full capacity within a time budget (a lost wakeup, permit leak, or stale-entry slot swallow fails instead of hanging). Full
tools::subagentslice: 602 passed;cargo fmtclean.Port notes: adapted to current main — the roster-route tests'
acquire_ownedcall sites converted to the new gate API;running_countis kept (still used by tests).Credits
Original implementation by @asto18089 (co-authored), hardened through five documented review follow-ups (governor wiring on the main path, pause liveness, lost-wakeup race, pinned queued acquire, direct stress tests).