Add heartbeat_max_test_duration to cap per-test heartbeat duration#397
Merged
Add heartbeat_max_test_duration to cap per-test heartbeat duration#397
Conversation
ianks
force-pushed
the
heartbeat-max-test-duration
branch
from
c0af2a3 to
344a97c
Compare
…ost-test reclamation A stuck test would heartbeat forever (since PR #384 removed the countdown), preventing other workers from reclaiming it via reserve_lost. Add --heartbeat-max-test-duration N (defaults to timeout*10) so the heartbeat thread stops ticking after N seconds per test. Once ticking stops the ZSET score goes stale and reserve_lost can steal the entry. The started_at timestamp is passed through the tick state from with_heartbeat so elapsed is measured from when the test actually started, not from when the heartbeat thread woke up (which could be up to 1 second late due to the @cond.wait(1) timeout causing a skewed stale threshold). Fixes #395
ianks
force-pushed
the
heartbeat-max-test-duration
branch
from
8b1a520 to
c321e91
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After #384 removed the heartbeat countdown (to fix the poisoning bug), stuck tests
ended up heartbeating forever -- other workers could never reclaim them via
reserve_lost. This PR adds a per-test heartbeat cap so the entry eventually goesstale and can be stolen.
What changed
--heartbeat-max-test-duration SECONDSCLI flag -- once a test has been runningfor this long, the heartbeat thread stops ticking. The ZSET score then goes stale
after
max_missed_heartbeat_secondsand another worker can pick it up.timeout * 10when heartbeat is enabled, so existing setups getreasonable behavior out of the box.
Timing gotcha
There was a subtle bug I hit while building this: the heartbeat thread can miss the
initial
:tickbroadcast (the condition variable fires before the thread callswait), so the first tick can land up to 1 second late. If you measure elapsed from"when the thread first woke up", the stale threshold ends up skewed by that 1 second
-- which can put it right at the moment the stuck test naturally finishes, leaving no
steal window.
The fix is to pass
started_at = Process.clock_gettime(CLOCK_MONOTONIC)through thetick state from
with_heartbeat, so the elapsed calculation is always anchored towhen the test actually started.
closes #395