Skip to content

fix(supervisor): commit a proven line before an ended turn's custody is read - #321

Merged
shtse8 merged 1 commit into
mainfrom
agent/aa-supervisor-idle-race-test-sync
Nov 16, 2026
Merged

shtse8 merged 1 commit into
mainfrom
agent/aa-supervisor-idle-race-test-sync

Conversation

@shtse8

@shtse8 shtse8 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #315: two flakes it introduced on main and in the merge queue.

Supervisor: a deferred line's turn lost its custody

A harness that answers Deferred can take and finish a turn before the pass returns. The idle pass then read the ended turn while the store held no open turn for its proven line:

  • an errored turn was followed up with opened_seq = None: the recovery note was skipped after the backoff, and has_open_turn stayed false for the whole backoff;
  • a completed turn was left open behind its own proof (the later duplicate offer opened it, and nothing closed it).

record_usage_with now commits what the harness proved before it reads or closes an ended turn's custody, and retries the pass when the store cannot record the proof.

Regression tests (errored_turns.rs): a vendor whose deliver writes, finishes and faults the turn but answers Deferred and proves the line. Both the errored and the completed case fail before the change (open-turn assertions) and pass after.

Tests: clock races

  • set_ms right after the first delivery could land before the worker installed its post-delivery timer, putting that timer past the new time with nothing to wake it. Time now moves only to a timer a worker is blocked on (RecordingClock::fire_next_timer), capped inside the window the backoff assertion allows.
  • Mail sent during the backoff could wake the worker before time reached the deadline minus one, so the awaited (deadline - 1, Some(deadline)) wait was never installed. Time now stands at that point before the mail arrives.

RecordingClock moves to the shared tests/support; spiron_tests.rs gets the same two changes.

Verification

  • cargo test -p anyagents-supervisor --test errored_turns and spiron_supervisor_recovers_from_transient_mid_turn_server_error_via_recovery: pass.
  • Temporary stress loops (removed before commit), run in parallel: 60 + 60 iterations of the terminal-backoff test, 12 of the growing-backoff test, 40 of each deferred-proof test, and 40 of the Spiron recovery test. All passed.
  • cargo clippy -p anyagents-supervisor -p anyagents --tests -- -D warnings and cargo fmt --check: clean.
  • git merge-tree --write-tree HEAD 261d211 ([review] feat(spiron): speak spiron.headless/v2 and run local Spiron without a keeper #280 head): clean.

…is read

A harness that answers `Deferred` can take and finish a turn before the
pass returns. The idle pass then read the ended turn while the store held
no open turn for its proven line: an errored turn was followed up with no
turn to carry (the recovery note was skipped, and the turn stayed closed
through its whole backoff), and a completed turn was left open behind its
own proof.

`record_usage_with` now commits what the harness proved before it reads
or closes an ended turn's custody, and retries the pass when the store
cannot record the proof.

Test clocks no longer race the supervisor: time moves only to a timer a
worker is blocked on, and mail that arrives during a backoff does so with
time already at its deadline.
@shtse8
shtse8 enabled auto-merge November 16, 2026 02:28
@shtse8
shtse8 added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 3dca938 Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant