feat(durable): add durable execution engine for v2

[mpangrazzi]

Stack

• Base: main
• Durable dashboard follow-up: feat(dashboard): surface durable execution attempts #265

What

Adds opt-in durable execution for Haystack 3.1 Pipelines and Agents. Executions are detached from the submit request and can survive process loss when backed by Redis.

Features

• Typed submit, inspect, cancel, resume, and reattachable SSE stream routes
• Explicit queued, running, waiting, completed, failed, and canceled lifecycle
• Checkpoints, progress, cooperative cancellation, application retries, and process recovery
• Pure storage-neutral reducer with fenced leases and cancellation-wins semantics
• In-memory reference store plus Redis 6.2+ storage with atomic transitions, Redis server time, revision-specific queues, bounded payloads, admission limits, and terminal retention
• Haystack 3.1 Pipeline and Agent recovery adapters
• Immutable deployment revisions and safe overwrite/undeploy guards while work is live
• Runtime/store health in /status, operational documentation, examples, and real-Redis CI coverage

Scope

This PR deliberately does not add durable A2A task execution or recovery. Existing A2A execution remains request-bound. Durable dashboard observability is isolated in stacked PR #265.

Reviewer guide

This is a large foundational change. A useful review order is:

1. src/hayhooks/durable/engine.py, models.py, and store.py: lifecycle, public projection, and persistence contract.
2. src/hayhooks/durable/redis.py: atomic Redis implementation, indexes, fencing, limits, and TTL behavior.
3. src/hayhooks/durable/context.py, runtime.py, and haystack.py: worker ownership, heartbeats, retries, checkpoints, thread shutdown, and Haystack recovery.
4. src/hayhooks/durable/fastapi.py: ownership, idempotency, REST status codes, and bounded SSE replay.
5. Hayhooks integration in server/app.py, server/utils/deploy_utils.py, and server/utils/module_loader.py: startup, dynamic deploy rollback, and live-work protection.
6. Docs and examples for the supported authoring and operations contract.

Please keep these invariants in view:

• The reducer is the only lifecycle authority. Stores only apply transition plans atomically.
• Redis time, leases, and fences prevent stale workers from committing.
• Cancellation wins every concurrent owned outcome.
• Nonterminal work is pinned to its immutable definition revision.
• Inputs, outputs, checkpoints, progress, and stream history are bounded.
• Owner mismatches are hidden as not found, and idempotency is owner-scoped.
• Stream chunks are display-only and retained until the execution reaches terminal state.
• Failed dynamic publication restores files, routes, registry state, and the previous durable deployment independently.

Pre-GA verification

• hatch run test:unit: 827 passed, 4 skipped, 26 deselected, 3 xfailed
• hatch run fmt-check and hatch run test:types
• hatch run docs:build --strict
• Real Redis integration, including process-kill recovery: 10 passed
• Base and durable wheels built and installed in fresh virtual environments
• Full-server SIGKILL recovery for waiting and running executions; SSE reattachment and cooperative cancellation verified
• Compose-backed Redis outage: /status degraded during interruption, persisted execution recovered on attempt 2, then health returned to Up
• Four-worker SSE soak: 12/12 terminal events and 12/12 final chunks
• Five 16-way submission bursts: 80/80 accepted and completed
• Dashboard tests, lint, and production build are covered in stacked PR feat(dashboard): surface durable execution attempts #265

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​redis@​8.1.0

View full report

[mpangrazzi]

Manual pre-GA failure and recovery canaries

These checks were run against a real Hayhooks server and Redis service, outside the unit/integration suites.

Hayhooks process killed while durable work was active

• Started the full Hayhooks CLI server with Redis-backed durable storage and a deterministic long-running pipeline.
• Submitted two executions: one already suspended at a durable checkpoint and one actively running.
• Opened SSE streams before the failure so client behavior across process loss was also exercised.
• Sent SIGKILL to the Hayhooks process. This bypassed graceful shutdown and left Redis as the only surviving execution state.
• Confirmed the existing SSE connections terminated, as expected when the server process disappears.
• Restarted Hayhooks with the same pipeline definitions, Redis database, and durable key prefix.
• Reconnected through the executions' stream URLs rather than submitting replacement work.
• The running execution was reclaimed after its abandoned lease expired and completed on attempt 2.
• The suspended execution remained waiting, accepted its resume input after restart, emitted its final result=41 stream chunk, and completed on its recovered attempt.

Result: both executions survived abrupt host-process loss and remained inspectable/reconnectable without resubmission.

---

Redis stopped during an active execution

• Started the full Hayhooks server with four durable worker slots against the Redis 6.2 service from examples/durable-compose.yaml.
• Submitted a long-running durable execution and waited until a worker owned it.
• Stopped the Redis container while leaving Hayhooks running.
• Queried /status during the outage and observed the server report Degraded; the durable deployments were unhealthy with ExecutionStoreError and a non-zero store error streak.
• Recreated the Redis service using the same Compose volume, preserving the execution data written before the outage.
• Kept the same Hayhooks process running and allowed its operational retry loop to reconnect to Redis.
• The interrupted execution was recovered and completed on attempt 2.
• Queried /status again and observed Up, all four worker slots healthy, and the store error streak reset to zero.

Result: Redis loss degraded health without losing persisted work, and both execution processing and health recovered when Redis returned.

---

SSE terminal/chunk race under concurrent workers

• Ran 12 durable executions concurrently against a full server configured with four worker slots.
• Each execution emitted a final display chunk immediately before committing its terminal result.
• In the first soak, all 12 streams received completed, but one stream missed its final chunk even though that chunk was present in Redis.
• This isolated the failure to SSE read ordering: terminal state could be observed after an empty chunk read, causing the stream to close before draining a chunk committed in between.
• Changed the stream loop to observe the execution control/terminal fence before reading chunks and added a deterministic regression for that ordering.
• Repeated the same 12-execution/four-worker soak.
• All 12 streams then received both their final chunk and their completed event.

Result: the canary found a real release-blocking stream race, and the repeated run verified the fix under concurrency.

---

Concurrent submission contention follow-up

• An earlier exploratory high-burst run produced an isolated HTTP 503, so a bounded follow-up was run with complete response capture.
• Used the real Redis-backed server with four worker slots and the normal nonterminal admission limit of 1,000.
• Submitted five rounds of 16 simultaneous POST requests, each with a unique idempotency key: 80 executions total.
• Recorded every HTTP response instead of allowing the client helper to hide the response body.
• Polled the stored execution controls until every accepted execution became terminal.
• All 80 requests returned HTTP 202, and all 80 executions completed; no 503 reproduced.

Result: the final contention run accepted and completed 80/80 executions across repeated concurrent bursts.

---

Cooperative cancellation on the real runtime

• Submitted a separate live durable execution against the Redis-backed full server.
• Requested cancellation while the execution was active.
• Allowed the pipeline to encounter its durable cancellation check rather than terminating the worker externally.
• Verified that the persisted public execution state reached terminal canceled and remained inspectable afterward.

Result: cancellation propagated through the running workload and committed the expected durable terminal state.

---

Durable A2A execution was intentionally not included in these canaries because it is outside the v2 scope of this stack.

[ArzelaAscoIi]

mighty PR! 👀

Looks good! I went through it with claude for me to digest it better 😅 I think all the comments are rather race conditions/things we should tackle later. Lets go!

[ArzelaAscoIi]

This is the one owned-outcome branch that doesn't check cancel_requested_at_ms. If a worker releases while cancellation is pending, this re-queues instead of finalizing to CANCELED — the run can sit reporting non-terminal state indefinitely. Should this mirror Suspend/ScheduleRetry and terminal-cancel here?

[ArzelaAscoIi]

This only catches ExecutionNotFoundError. A lease renewed between the zrange snapshot and this call raises InvalidExecutionTransitionError (same case claim(candidate=True) already handles a bit further down), and that will abort the rest of this maintenance batch — leaving every other already-due lease unprocessed. Worth catching it here too (and in MemoryExecutionStore.maintain).

[ArzelaAscoIi]

Same gap as the Redis store's maintain(): only ExecutionNotFoundError is caught. claim() above (line 260) already catches (ExecutionNotFoundError, InvalidExecutionTransitionError) for the equivalent race — this loop should too, otherwise one raced lease recovery kills the whole maintenance batch.

[ArzelaAscoIi]

Heartbeat writes only lease_expires_at_ms here instead of going through _apply_plan, so the rest of plan.next_control (version, updated_at_ms, etc.) that decide() computed is never persisted. The caller gets a plan claiming those fields changed, but a subsequent read() won't reflect it — and it diverges from MemoryExecutionStore, which applies the full plan for every command. Intentional perf shortcut, or should this call _apply_plan?

[ArzelaAscoIi]

thread_done from start_daemon_thread is dropped (_) here — this nested Pipeline thread isn't registered in _draining_runs the way the top-level runner thread is. On cancellation/shutdown it keeps running unobserved, and a later retry can race a genuine concurrent duplicate execution. Should this track/drain the thread the same way the top-level runner does?

[ArzelaAscoIi]

Rollback failures here (and at line 1012) are swallowed to a log line, then the original error is re-raised as-is. The caller only learns the deploy failed, not that the previous pipeline registration/routes may not have been fully restored (old_published = False). Given the PR description's claim that failed publication "restores files, routes, registry state, and the previous durable deployment independently," should a partial-restore failure surface differently — e.g. a distinct error or a flag callers can check — instead of looking identical to a clean rollback?