Skip to content

test(callbacks): stabilize deadline-release fixture against heartbeat scheduling race #1141

Description

@dennisonbertram

Work type

Bug

Problem

Three callback deadline-release fixtures use a 40 ms lease and wait for a channel that closes only when the heartbeat reaches ExtendLease. Under CI load the independent deadline may correctly cancel admission first; the heartbeat sees cancellation and exits without calling ExtendLease. Production terminalizes the attempt correctly, but the fixture waits for an event that will never happen.

Evidence: #1138 fast CI failed TestCallbackManagerDeadlineReleaseHonorsAttemptBound with deadline did not cancel bounded admission. Current main 68a82ddf and failing head have identical callback/test blobs. Focused normal/race ×100 passed locally, confirming an intermittent baseline fixture race.

Scope

Test/docs only. Stabilize:

  • TestCallbackManagerDeadlineReleaseRearmsSingleOwner
  • TestCallbackManagerDeadlineReleasePersistsSafeRetryReason
  • TestCallbackManagerDeadlineReleaseHonorsAttemptBound

Use the established #1121 pattern: one-second fixture lease; wait for admitted entry, heartbeat deadline entry, and cancellation-aware starter cancellation before unblocking renewal; retain the durable-state assertions.

Out of scope

  • Callback production behavior or retry policy changes.

Acceptance criteria

  • Fixtures prove actual deadline cancellation rather than a heartbeat scheduling side effect.
  • Attempt-bound case proves failed state, attempt 1, unchanged run ID, empty next-attempt/token/lease, and callback admission unavailable.
  • Focused normal/race three-test group ×20 and ./scripts/test-regression.sh pass.
  • Hosted fast and race checks pass.

Impact/rollback

Fixture and evidence only; no runtime/API/persistence rollout impact.

Test-first plan

First make the cancellation-aware assertions deterministic with the longer lease and explicit gates; then retain all durable-state assertions and run normal/race/full regression.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions