Skip to content

feat(quota): background poll keeps idle account quota fresh - #56

Merged
ualtinok merged 1 commit into
cortexkit:mainfrom
iceteaSA:feat/quota-background-poll
Aug 10, 2026
Merged

feat(quota): background poll keeps idle account quota fresh#56
ualtinok merged 1 commit into
cortexkit:mainfrom
iceteaSA:feat/quota-background-poll

Conversation

@iceteaSA

@iceteaSA iceteaSA commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Why

Quota is push-primary (x-codex-* headers / WS codex.rate_limits frames), so a fallback account that is not currently being routed to shows stale or missing quota until the operator manually runs /openai-quota. The sibling anthropic-auth plugin keeps idle accounts fresh with a background poll; this brings the same behavior here — without request spam from multiple concurrent opencode processes, and without letting any process publish stale data over fresh.

What

Background poll

  • core/background-quota-refresh.ts — singleton 5-minute timer (±30 s jitter per process), unref'd so it never keeps the process alive, idempotent start, stopped from both the plugin-level and auth-loader dispose paths.
  • Overlap-safe: the timer holds the in-flight run's promise and drops a tick that fires while the previous run is still executing (skip, not queue) — no concurrent token refreshes or duplicate wham calls from a slow tick.
  • Each tick calls the existing refreshAllQuota with respectBackoff: true plus a 4-minute freshness gate (skipFresherThanMs). Rides the 15 s abort timeout + quota-channel logging from fix(quota): bound and log the active quota fetch path #52. Deliberate deviation from the sibling: no immediate first run — the boot seed covers startup; first poll fires at the jittered 5-minute mark.

Cross-process dedup (no locks)

  • The freshness gate consults BOTH the in-memory QuotaManager snapshot AND the machine-global sidebar state file (written on every quota push by any process, read once per tick, tolerant reader, fail-open on missing/corrupt/malformed checkedAt). Whichever process polls first refreshes the file; every other process's tick inside the window sees the fresh checkedAt and skips. Worst case under jitter collision is 2 concurrent fetches, never N.
  • An all-skipped background tick performs no sidebar write (nothing new to publish).

Stale-write clobber fix (machine-state sidebar writes)

  • Previously any process's sidebar write replaced the machine-global quota rows wholesale from its own memory — an idle process with a stale cache could overwrite fresher rows another process just wrote (and thereby corrupt the dedup gate reading those rows). setSidebarMachineState now freshness-merges per-account inside the existing compare-retry callback: for main and each fallback (matched by id), the side with the fresher finite checkedAt wins; invalid/missing loses to valid; both missing → incoming wins. activeId/route keep last-writer semantics; activeRouting merge unchanged.
  • Snapshot-level quota.checkedAt added: a complete authoritative frame may retire every window (no primary.checkedAt left), and without a timestamp on the snapshot itself the freshness merge would resurrect the retired window from disk.

Unchanged: the manual /openai-quota command path (no gate, always writes) and the boot seed. No new config keys, commands, or TUI changes.

Verification

  • RED-first throughout: freshness gate (expected 1 wham call, got 3), cross-process skip (empty memory + fresh file: expected 0 calls, got 3), stale-write clobber (fresh disk rows expected retained, got stale 90/40), all-skipped no-write (expected 0 writes, got 1), tick overlap (expected 1 run, got 2), window-retirement tombstone.
  • Review (gemini-3.1-pro): APPROVE 0 must / 1 should — lifecycle, gate ordering (skip before token refresh), backoff interaction, restart-with-empty-cache behavior, command/boot-seed invariance verified; the should (jitter bounds untested) folded as a test pinning random()=0/1 → ∓30 s.
  • cubic P2 (tick overlap) fixed in the same commit; thread resolved.
  • Gates: build ✓ · tsc ✓ · full suite 692/0 ✓ · biome clean.

@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch from 1399ceb to e67f58c Compare July 21, 2026 20:31

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/opencode/src/index.ts
@ualtinok

Copy link
Copy Markdown
Contributor

Thanks. Background quota refresh is useful, but this needs revision before merge:

  1. Rebase onto current main (v0.4.2), which includes bounded quota/OAuth fetches from fix(quota): bound and log the active quota fetch path #52.
  2. Replace sidebar freshness check-then-fetch with an atomic cross-process refresh claim. The sidebar write lock serializes writes after fetching and does not prevent N stale processes from issuing N quota calls.
  3. Bind shared freshness to the current main-account identity and fallback token fingerprint. Quota from a prior login must not suppress a refresh after account switch or re-login.
  4. Route expired main-token refresh through the existing main refresh lease rather than calling the provider directly.
  5. Make lifecycle instance-safe: disposing one loader must not stop another loader's poller, and stop must define behavior for an in-flight run.
  6. Include secondary-window timestamps in freshness selection, and add multi-process, account-switch, re-login, secondary-only timestamp, and disposal tests.

Please rerun the complete gate after these changes.

@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch from 823e46a to 3751617 Compare July 22, 2026 18:38
@ualtinok

Copy link
Copy Markdown
Contributor

Thanks for the revision. Rebase and main-refresh lease use are now correct.

Remaining blockers:

  1. Freshness must use live main identity and fallback token identity. Loader-captured main IDs and internal fallback IDs can reuse a prior login's fresh cache. Add stale in-memory re-login coverage, not only sidebar account-switch coverage.
  2. The cross-process refresh lock has a 30-second non-renewing TTL, but serial account refresh can exceed it. Renew the lock or use a bounded claim whose lifetime covers the entire run, with an over-TTL concurrency test.
  3. The poller remains process-global. One loader overwrites another callback and either disposal stops the shared poller. Make it per-loader or reference-counted, and add two-loader disposal plus in-flight-stop tests.
  4. Sidebar freshness merge must consider secondary-window checkedAt, not only primary or snapshot-level timestamps. Add a newer-secondary ordering regression.

Please rerun the full gate after these changes.

@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch from 3751617 to 578efd0 Compare July 22, 2026 20:11
@ualtinok

Copy link
Copy Markdown
Contributor

Thanks for the revision. The per-loader lifecycle and in-flight stop work are now covered.

Three blockers remain:

  1. Background main quota writes still store no account identity, and peekMainForPolicy accepts an unknown identity. A background result for the old account can therefore remain fresh after re-login. Fallback in-memory freshness is likewise keyed only by internal ID. Persist and require live identity metadata for both cache paths.
  2. The cross-process background claim still has no renewal. A serial refresh can outlive its TTL and allow a second process to begin the same work. Use the lock's renewal support and add an over-TTL contention regression.
  3. Sidebar quota freshness is account-level, not per-window. A snapshot with a newer primary can replace another snapshot's newer secondary. Merge primary and secondary windows independently and add the crossed-timestamp regression.

Please rerun the full gate after these fixes.

@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch 2 times, most recently from 885fd94 to 09f4d2f Compare July 23, 2026 06:55
@iceteaSA

Copy link
Copy Markdown
Contributor Author

Fixed in 09f4d2f. All three blockers:

  1. Identity metadata on both cache paths. Main: setMain stores the identity and peekMainForPolicy(identity) drops on a genuine switch. Fallback: identity now rides QuotaEntry.accountId (converged onto fix(routing): skip known-exhausted accounts at admission #57's mechanism — see below); peekFallbackForPolicy(id, identity?) returns null on a known mismatch, fail-open when unknown. refreshAllQuota threads the live main identity (storage.mainAccountId, re-loaded at call start) and the fallback's live accountId, so a prior login's cache can't suppress a refresh after re-login. Added stale-in-memory re-login coverage for both, not just sidebar account-switch.

  2. Lock renewal, not just a longer TTL. The background claim now acquires with renew:true (extracted to acquireBackgroundRefreshLock), re-arming the TTL until release(); TTL also raised to 120s. Over-TTL contention test: holder keeps the lock across a simulated 125s, a contender is locked out, release discriminator confirmed.

  3. Per-window merge. setSidebarMachineState now merges primary and secondary independently by checkedAt, gated on matching account identity (differing/unknown identity whole-picks the fresher snapshot so two accounts' windows are never combined), and honors window-retirement (a fresher snapshot's absent slot means the wire dropped that window). Crossed-timestamp regression added: a snapshot with a newer secondary is no longer clobbered by another's newer primary.

Full gate green (748/0). The fallback-identity mechanism here is byte-identical to #57's (QuotaEntry.accountId) so the two branches share one identity surface.

@ualtinok

Copy link
Copy Markdown
Contributor

Thanks, the live-identity binding and renewable background claim are now addressed.

One blocker remains in the per-window merge: buildSidebarMachineState() writes only the snapshot-level checkedAt, but mergeQuotaByWindow() compares primary.checkedAt and secondary.checkedAt. Those window stamps are absent on normal writes, so a concurrent incoming snapshot still wins both slots and can replace a newer secondary window.

Propagate the entry timestamp onto each present window when serializing the sidebar state, or make finiteWindowCheckedAt() fall back to the enclosing snapshot timestamp. Add the crossed-timestamp regression using the same normal writer path, then rerun the full gate.

@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch from 09f4d2f to 4d443f0 Compare July 24, 2026 13:42
@iceteaSA

Copy link
Copy Markdown
Contributor Author

Fixed in 4d443f0 — both halves, so the per-window merge engages regardless of which generation wrote the file:

  1. Read-side fallback: finiteWindowCheckedAt(window, fallback?) now falls back to the enclosing snapshot's checkedAt when the window carries no finite stamp of its own; mergeQuotaByWindow passes each side's snapshot stamp through. Files written by older builds (no window stamps) now merge correctly instead of letting the incoming snapshot win both slots.
  2. Write-side stamping: buildSidebarMachineState() stamps each present window with the entry's checkedAt when the window lacks its own (main + every fallback). Absent windows are never fabricated, so retirement semantics are untouched.

Regression added through the normal writer path as requested: two identity-matched writes with crossed timestamps (A: both windows, snapshot T2, no window stamps — mirroring old files; B: stamped windows at T1 < T2). RED on the pre-fix code — B clobbered A's newer windows (Expected: 50, Received: 60) — green with the fix; A's newer primary and secondary both survive.

Retirement (absent slot in a fresher snapshot never resurrects) and identity-mismatch whole-pick behavior verified unchanged. Full gate: build + typecheck + 749/0 tests + biome clean.

@ualtinok

Copy link
Copy Markdown
Contributor

Verified both halves at 4d443f0. The read-side fallback is real (sidebar-state.ts:566 finiteWindowCheckedAt with a snapshot fallback, threaded through freshestWindow and mergeQuotaByWindow), and the write-side stamping is real and on the production path (index.ts:476 stampWindowCheckedAt, applied to every present main and fallback window inside buildSidebarMachineState, which writeMachineSidebarState calls). The per-window merge now actually engages. Lock renewal (renew: true, 120s TTL held across the refresh with release in finally) and the identity binding across memory, sidebar, and the freshness gate all check out.

One thing to tighten before merge, and it is the test rather than the code.

The writer-path test at sidebar-state.test.ts:1723 reds only on a full revert, not on either half alone. Because write-side stamping is in effect, A's unstamped windows are stamped T2 on the way to disk, so the merge compares window T2 against window T1 and the read-side fallback is never reached. Revert only the write-side and the read-side fallback carries it; revert only the read-side and the stamps carry it. The comment says "A's windows carry no per-window stamps — mirroring files written by code that only stamped the snapshot-level checkedAt", but that describes the QuotaManager input, not what lands on disk.

That matters because the read-side fallback exists for exactly one case: a file written by a pre-fix build. Nothing currently exercises that shape end-to-end. :1491 covers true crossed windows but hand-builds the stamps and bypasses the writer entirely.

Please add one test that seeds the file directly with old-shape content (both windows present, no window stamps, snapshot checkedAt only), then writes through buildSidebarMachineStatesetSidebarMachineState with crossed per-window stamps, and asserts both windows survive. That reds on either half alone and pins the migration path the fallback was written for.

Nothing else outstanding — with that test in place this is ready.

Quota only refreshed for an account that served a request, so a fallback
that had not been routed to in a while showed stale numbers - and those
numbers are what routing and the killswitch decide on.

An unref'd five-minute jittered poll now refreshes every account. A
four-minute freshness gate skips accounts that recent traffic already
covered, and the gate reads the shared sidebar file rather than local
memory so N concurrent processes do not each pay for the same refresh.
A cross-process lock with an auto-renewing lease keeps that to one
refresher at a time.

Snapshots merge per window by newest checkedAt instead of last-writer-
wins, so an idle process cannot write its stale view over a fresher one.
Each window carries its own stamp for that comparison, and a snapshot-
level stamp remains as a fallback for files written before this change.

Two tests cover the halves separately, because no single one can cover
both: buildSidebarMachineState derives the per-window stamp and the
snapshot stamp from the same value, so reverting the stamping leaves the
fallback supplying an identical timestamp. The first seeds an old-shape
file directly - both windows present, no per-window stamps - and asserts
they survive a write carrying crossed stamps, which is the migration
path the fallback exists for. The second pins why the stamping matters:
the merged snapshot stamp is the max across both windows, so a winning
window that lands unstamped beside a fresher-stamped sibling inherits
that sibling's freshness on the next read and can beat a write that is
genuinely newer than it.
@iceteaSA
iceteaSA force-pushed the feat/quota-background-poll branch from 4d443f0 to 4063e57 Compare August 10, 2026 18:02
@iceteaSA

Copy link
Copy Markdown
Contributor Author

Pushed 4063e57 — 751 pass / 0 fail, test-only, production byte-identical.

Your diagnosis of :1723 was exactly right, and chasing it turned up something worth reporting: the property you asked for is unachievable in a single test, and I can show why rather than just assert it.

Why one test cannot red on either half alone. buildSidebarMachineState derives the per-window stamp (:503/:507) and the snapshot-level checkedAt (:502) from the same mainEntry.checkedAt. So for anything entering through the writer, the stamp and the fallback are the same number — revert the stamping and finiteWindowCheckedAt hands back an identical value. They are architecturally redundant on the incoming side, which is precisely why :1723 only reds on a full revert.

I tried two ways to break that redundancy before concluding it:

  1. Divergent stamps on the existing side — seed a file whose windows carry genuinely different stamps, then write through the writer. No effect: the redundancy is about what the incoming side carries into the merge, not what it reads from disk.
  2. Max-inflation at generation three — this one is real, and it is the answer to "what is write-side stamping actually for".

On (2). mergeQuotaByWindow computes the merged snapshot stamp as max(primary.stamp, secondary.stamp). Without write-side stamping, a winning incoming window lands on disk unstamped beside a sibling carrying a fresher stamp. On the next read that unstamped window falls back to the snapshot value — the max — and is judged fresher than it truly is. A subsequent write that is genuinely newer than that window then loses a merge it should win.

So stampWindowCheckedAt is load-bearing, but its effect is only observable two merges downstream, and a test cannot reach it through the writer because the writer is what makes stamp and fallback equal.

Two tests, covering the halves separately:

  • Read-side (:1815) — the one you asked for. Seeds an old-shape file directly (both windows, no per-window stamps, snapshot checkedAt only), writes through buildSidebarMachineStatesetSidebarMachineState with crossed stamps, asserts both windows survive. Reds on read-side revert: expected 50, received 60.
  • Write-side rationale (:1889) — constructs the disk state that merge build(deps-dev): bump @earendil-works/pi-tui from 0.79.5 to 0.79.9 #1 would leave behind with stamping absent, and shows the inflated window winning a merge it should lose. Both arms asserted: inflated state resolves wrongly (80), correctly-stamped control resolves rightly (10).

Being straight about the limitation: the second test demonstrates the consequence of removing write-side stamping by constructing its output state — it does not gate the write-side hunk itself, and it reds on read-side revert rather than write-side. I would rather say that than let the pair look like it delivers the property you asked for.

If you want the write-side genuinely gated, the change is in production rather than tests: buildSidebarMachineState would need to stop deriving both stamps from one value. That is a larger change than this PR should carry, and I did not want to make it unilaterally.

:1723 left in place — it still covers the writer path with both sides going through buildSidebarMachineState, which neither new test does.

@ualtinok

Copy link
Copy Markdown
Contributor

Verified at 4063e572. The new test seeds the file with genuine pre-fix shape — both windows present, no per-window stamps, snapshot checkedAt only — then writes through buildSidebarMachineStatesetSidebarMachineState with crossed stamps, and asserts each slot resolves independently. That pins the read-side fallback on its own, which the earlier test could not do. The second test covering snapshot-max inflation across generations is a good catch that I had not asked for.

751 tests pass locally on the branch. Ready to merge from my side.

@ualtinok
ualtinok merged commit 0676a12 into cortexkit:main Aug 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants