Order flow-run heartbeats with state transitions

[desertaxle]

closes #21932

Depends on #22948

This PR orders flow-run heartbeat admission with server-authoritative state requests, preventing a heartbeat from being enqueued after the server accepts a terminal transition.

Details

• Builds and validates heartbeat events before admission, then gates only the in-memory enqueue with each state request.
• Drains an admitted enqueue before sending the state request and permanently closes admission when the server returns a final state.
• Reopens admission after WAIT and accepted nonfinal responses.
• Uses the proposal module's narrow request interface so each retry is ordered without wrapping a partial client.
• Applies the same contract to sync and async engines, with regression coverage for both race orderings and shutdown.

Supersedes #22897 and carries forward Devin's initial work.

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this is a complex change, a maintainer has confirmed the proposed approach on the linked issue.
• If this pull request adds or changes functionality, it includes tests or explains why tests are not needed.
• If this pull request changes user-facing behavior, it updates documentation or explains why documentation is not needed.
  • The documented automation pattern already describes the intended behavior; this changes client heartbeat ordering only.
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[codspeed-hq]

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing fix/flow-run-heartbeat-ordering (18c251d) with flow-run-state-proposals/foundation (582e5db)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e024e4fa15

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Await heartbeat delivery before terminal requests

When the events worker is busy or reconnecting, _enqueue_prepared_event only places the heartbeat in QueueService via put_nowait; releasing this gate therefore allows the terminal state HTTP request to be processed while the heartbeat is still queued for network delivery. If the server observes Completed first and the delayed heartbeat afterward, the documented proactive zombie-flow pattern can clear its bucket on completion and then open a new bucket from the late heartbeat, causing alerts or crash actions against an already-completed run. Gate the request on an acknowledgement/checkpoint of actual heartbeat delivery rather than only its in-memory enqueue.

AGENTS.md reference: src/prefect/AGENTS.md:L23-L23

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b0f7c5ebe

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve the public state-proposal aliases

Removing the propose_state and propose_state_sync imports also removes those non-underscored attributes from prefect.flow_engine, so downstream imports or patches now raise AttributeError. This already causes all eight TestFlowRunNameSetBeforeRunningEvent cases to fail at tests/test_flow_engine.py:116 and :133; keep compatibility aliases while routing engine internals through the new proposer wrappers.

AGENTS.md reference: src/prefect/AGENTS.md:L5-L7

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 79a319e2b9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Make async lock ownership cancellation-safe

When an async flow task is cancelled via asyncio.Task.cancel() while this worker call is waiting behind an in-flight heartbeat enqueue, the AnyIO CancelScope does not defer that direct asyncio cancellation: the coroutine unwinds with acquired == False, but the worker continues and can later acquire _gate with no code left to release it. Subsequent heartbeat admission or session.aclose() can then hang indefinitely, so the ownership handoff must remain observable and release the lock even after direct task cancellation.

AGENTS.md reference: src/prefect/AGENTS.md:L23-L25

Useful? React with 👍 / 👎.