Repository navigation
fix(supervisor): commit a proven line before an ended turn's custody is read - #321
Merged
Merged
Conversation
…is read A harness that answers `Deferred` can take and finish a turn before the pass returns. The idle pass then read the ended turn while the store held no open turn for its proven line: an errored turn was followed up with no turn to carry (the recovery note was skipped, and the turn stayed closed through its whole backoff), and a completed turn was left open behind its own proof. `record_usage_with` now commits what the harness proved before it reads or closes an ended turn's custody, and retries the pass when the store cannot record the proof. Test clocks no longer race the supervisor: time moves only to a timer a worker is blocked on, and mail that arrives during a backoff does so with time already at its deadline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #315: two flakes it introduced on main and in the merge queue.
Supervisor: a deferred line's turn lost its custody
A harness that answers
Deferredcan take and finish a turn before the pass returns. The idle pass then read the ended turn while the store held no open turn for its proven line:opened_seq = None: the recovery note was skipped after the backoff, andhas_open_turnstayed false for the whole backoff;record_usage_withnow commits what the harness proved before it reads or closes an ended turn's custody, and retries the pass when the store cannot record the proof.Regression tests (
errored_turns.rs): a vendor whosedeliverwrites, finishes and faults the turn but answersDeferredand proves the line. Both the errored and the completed case fail before the change (open-turn assertions) and pass after.Tests: clock races
set_msright after the first delivery could land before the worker installed its post-delivery timer, putting that timer past the new time with nothing to wake it. Time now moves only to a timer a worker is blocked on (RecordingClock::fire_next_timer), capped inside the window the backoff assertion allows.(deadline - 1, Some(deadline))wait was never installed. Time now stands at that point before the mail arrives.RecordingClockmoves to the sharedtests/support;spiron_tests.rsgets the same two changes.Verification
cargo test -p anyagents-supervisor --test errored_turnsandspiron_supervisor_recovers_from_transient_mid_turn_server_error_via_recovery: pass.cargo clippy -p anyagents-supervisor -p anyagents --tests -- -D warningsandcargo fmt --check: clean.git merge-tree --write-tree HEAD 261d211([review] feat(spiron): speak spiron.headless/v2 and run local Spiron without a keeper #280 head): clean.