Hayhooks V2: durable execution for pipelines and A2A agents

[mpangrazzi]

Scope: Haystack 3 only.

Stack

Merge in order: #253, then #258, then #259. PR #258 adds durable A2A trace correlation; PR #259 adds its dashboard classification and presentation.

Durable execution and managed A2A Agents

This PR adds durable execution for Haystack 3 Pipelines and Agents. It accepts typed work, persists it before returning, runs it outside the request, and recovers it from safe checkpoints after a worker or process disappears.

Highlights

• Crash recovery for native Haystack 3 Pipelines and Agents: Redis records plus explicit Pipeline or Agent checkpoints resume work after a worker or process disappears.
• Safe at-least-once delivery: fenced renewable leases keep one active owner while abandoned work is recovered automatically.
• Checkpoint-aware retries: bounded retries continue from the latest checkpoint instead of repeating completed upstream Pipeline work.
• Built-in interaction: typed inspection, progress, wait/resume, terminal results, idempotent submission, and cooperative cancellation.
• Live SSE chunk streaming: every execution exposes a reattachable GET /{pipeline}/executions/{id}/stream endpoint carrying best-effort display chunks at token rate plus the terminal projection. Chunks live in a bounded Redis stream deliberately outside the durable fence, so a dropped token can never fail or replay a run.
• Revision-safe rollouts: an exact durable_revision gate prevents incompatible queued or waiting work from resuming against changed code.
• First-class long-running A2A Agents: durable task projection covers progress, input-required continuation, completion, failure, cancellation, and Redis-backed recovery.
• Portable FastAPI integration: applications can own a DurableRuntime, configure it with DurableSettings, and include a typed durable APIRouter with their existing authentication dependencies.
• Focused operational footprint: purpose-built for moderate-scale durable Haystack workloads without introducing a separate general-purpose workflow platform.

See Hayhooks durable engine and Temporal for the capability and tradeoff comparison.

Portability and FastAPI integration

The durable engine can be embedded through hayhooks.durable without depending on Hayhooks server startup. Its public surface includes DurableRuntime, DurableSettings, DurableDeployment, ExecutionStore, ExecutionStoreProvider, the built-in memory and Redis providers, durable contracts and exceptions, and create_durable_router(); Redis remains optional at import time.

An application creates its runtime and deployment, starts and closes the runtime in its FastAPI lifespan, and includes the router under any prefix. The router exposes typed submit, inspect, cancel, resume, and stream operations for one deployment. A normal sync or async FastAPI dependency can return a stable owner ID, so the host keeps control of authentication and authorization. Passing owner_id_dependency=None explicitly selects unscoped access.

Hayhooks now consumes that same public router instead of maintaining a second REST implementation. Its internal shim only installs and removes per-deployment routes safely. REST, A2A, and MCP lifespans operate on the runtime instance they own rather than the process-global singleton.

Standalone runtimes own and start only their attached deployments; they do not inspect the process-global pipeline registry or require private server-loader flags. Runtime, deployment, provider, and store use one settings snapshot, conflicting settings fail before deployment, and provider replacement is locked once a provider or deployment candidate exists. The runtime owns provider shutdown.

See Embedding the runtime for the complete authenticated FastAPI example, lifecycle contract, unscoped-access warning, and Redis configuration.

Diff breakdown for reviewers

The PR is broad because it ships the engine together with recovery tests, runnable examples, operations documentation, A2A integration, and the portable FastAPI adapter. Of the 15,395 additions, 7,726 (about 50%) are tests, examples, or documentation.

Area	Files	Diff	What to focus on
Source	45	+7,584 / -612	Reducer/store correctness, chunk log, runtime ownership, FastAPI adapter, SSE stream route, deployment lifecycle, A2A projection, and recovery
Tests	31	+5,545 / -303	Contracts, chunk-log semantics, races, crash recovery, Redis integration, FastAPI ownership, streaming, A2A recovery, and lifecycle cleanup
Documentation	12	+918 / -123	Public API, embedding example, streaming reference, operating boundaries, configuration, and design rationale
Runnable examples	12	+1,263 / -4	End-to-end Pipeline, Agent, streaming chat, A2A, retry, approval, and process-recovery usage
CI and packaging	7	+85 / -12	Haystack 2/3 matrix, Redis service, durable extra, and docs checks
Total	107	+15,395 / -1,054

Suggested review order:

1. src/hayhooks/durable/engine.py, backend.py, store.py, redis.py, and reference.py for lifecycle, storage, and chunk-log correctness.
2. manager.py, context.py, adapters.py, runtime.py, settings.py, and fastapi.py for execution, configuration, Haystack boundaries, the public HTTP adapter, and the SSE stream route.
3. server/durable/routes.py, server/utils/deploy_utils.py, and the REST/A2A/MCP lifespans for runtime ownership and dynamic route lifecycle.
4. server/a2a/durable_executor.py and redis_task_store.py for the A2A-specific projection and recovery layer.
5. Tests for the corresponding contract and failure cases; examples/durable_chat_with_website and its tests are the streaming reference material.

Supported features

Durable REST execution

• An ordinary BasePipelineWrapper implements run_durable() or run_durable_async(). Hayhooks invokes that method for each durable execution with a DurableContext and typed Pydantic request.
• The wrapper calls context.run_pipeline() / context.run_pipeline_async() for checkpointed Pipeline work, or context.run_agent() / context.run_agent_async() for Agent work.
• Typed wrappers expose:
  • POST /{pipeline}/run-durable
  • GET /{pipeline}/executions/{id}
  • GET /{pipeline}/executions/{id}/stream
  • POST /{pipeline}/executions/{id}/cancel
  • POST /{pipeline}/executions/{id}/resume
• Submission validates and persists input before returning. Idempotency-Key replays the same operation safely and rejects reuse with different input.
• Executions provide bounded public progress, wait information, terminal results, and safe error details.
• The engine supports bounded retries, cooperative cancellation, typed wait/resume, terminal retention, and optional owner-isolated access through a trusted FastAPI dependency or Hayhooks owner header.
• Durable wrappers declare durable_revision, allowing queued and waiting work to be checked against checkpoint-relevant deployment code.

Pipeline checkpoints

Pipeline wrappers opt in through context.run_pipeline(..., checkpoint_at=[...]) or context.run_pipeline_async(..., checkpoint_at=[...]).

For every named boundary, Hayhooks asks Haystack to stop at a public Breakpoint and immediately persists the returned PipelineSnapshot before that component executes. If Haystack exposes a snapshot with a PipelineRuntimeError, Hayhooks persists that snapshot too.

On recovery, Hayhooks rebuilds the PipelineSnapshot, passes it back to the Pipeline, and starts from the saved scheduler state. Haystack skips the already-completed upstream component visits; work after the last checkpoint is replayed.

Agent checkpoints

Managed Agents use the public Haystack hook surface. Hayhooks installs synchronous and asynchronous versions of these hooks for the active durable execution:

Hook	Durable behavior
before_run	Restores the persisted serializable State while keeping fresh per-run tools and hook context.
before_llm	Checks for cooperative cancellation before every model call.
after_tool	Saves an Agent state checkpoint after a tool-result batch when the Agent will continue to another model step, then checks cancellation.
on_exit	Saves application-adjusted state when an application exit hook requests continue_run.
after_run	Saves a final checkpoint. A recovered execution returns this saved Agent result without another model call.

The serialized Agent checkpoint excludes live tools and hook_context; the current deployment recreates them for the recovered run. Checkpoints and progress are saved together as a durable execution transition.

Redis recovery

• Redis is the default durable store; the in-memory store provides the same contract for local development and tests.
• Each deployment keeps its own controls, opaque payloads, one bounded chunks stream per execution, and two indexes: runnable for queued work and lease-expiry for active claims.
• Workers claim due work with monotonic fences and renewable leases. Redis TIME, optimistic transactions, and fenced transitions keep ownership safe across replicas.
• Lease maintenance requeues abandoned work. Redis TTL retains terminal records, chunk logs, and idempotency bindings for the configured window.
• Health reports durable nonterminal, runnable, and lease_expiry counts.

Streaming chunks

• DurableContext.stream_chunk() (and stream_chunk_sync() for worker threads) append one display chunk outside the durable fence: a single XADD that never contends with the lease heartbeat.
• Chunks are best-effort display data, not durable state. An oversized chunk (capped by HAYHOOKS_DURABLE_MAX_STREAM_CHUNK_BYTES, 64 KB default) or a backend blip drops that chunk and logs it, never failing or replaying the execution.
• HAYHOOKS_DURABLE_MAX_STREAM_CHUNKS bounds the log per execution (10 000 default); 0 disables chunk production while leaving the endpoint working.
• SSE events carry the stream entry ID as id: and the producing attempt in the payload. Reconnecting clients resume from Last-Event-ID without a gap; a retried attempt re-streams from its checkpoint, so clients reset their buffer when attempt increases.
• The stream ends with a terminal completed/failed/canceled event carrying the same projection as the inspect route. A break after headers are sent ends in an SSE error event instead, which clients treat like a dropped connection.
• The chunk log shares the execution's terminal TTL, including stale appends from workers that lost their lease, so it cannot outlive its execution's retention window.
• See the streaming reference and examples/durable_chat_with_website, which detaches mid-answer, reattaches, and survives a process restart.

Managed long-running A2A Agents

• A Haystack 3 Agent in A2APipelineWrapper is exposed as a durable A2A Agent. Hayhooks supplies the durable worker, queue, execution record, checkpointing, progress projection, and Redis integration.
• The durable execution record supplies A2A progress, input-required, completion, failure, and cancellation state. A follow-up A2A message resumes work while the execution is waiting for input.
• Redis-backed A2A task storage persists task-to-execution bindings and task snapshots. Startup repairs active snapshots with compare-and-set version fencing; GetTask and list requests project the latest durable state directly.
• A2A task retention and durable execution retention are independently configurable.

Execution model

The lifecycle is a pure reducer over a compact execution control record:

queued -> running -> completed | failed | canceled
              |\
              | -> queued  (retry or expired lease)
              | -> waiting -> queued  (resume)

The store atomically persists each reducer plan and its derived Redis indexes. The durable manager owns worker polling, lease heartbeats, retries, and shutdown draining; Haystack adapters own the Pipeline snapshot and Agent hook integration. Display chunks are the one deliberate exception: they bypass the reducer entirely.

See the durable operations guide.

Validation

• Complete local suites: Haystack 3: 820 passed; Haystack 2: 718 passed (expected skips and xfails excluded).
• Streaming additions are covered by the durable unit suites (execution, FastAPI, reference contract) and Redis integration tests, including cursor replay, attempt tagging, the chunk-log kill switch, TTL retention of stale appends, and a real-process concurrent-streaming integration test gated on OPENAI_API_KEY.
• Ruff, type checking, and the strict documentation build: passed locally.
• GitHub Actions validates Python 3.10–3.14, the Haystack 2/3 matrix, dashboard tests, documentation, and packaging.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​redis@​8.1.0

View full report

[socket-security]

All alerts resolved. Learn more about Socket for GitHub.

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report

[sjrl]

Just FYI this is fixed in Haystack 3.1 from PR deepset-ai/haystack#12185

[sjrl]

Not a big deal, but editing Hooks this way does skip hook validation. In Haystack 3.1 you could use the Agent.clone method and pass in your new hooks there as an alternative. Only if you want the init to fire again to do the validation checks.

[mpangrazzi]

closing in favour of #264