Hayhooks V2: durable execution for pipelines and A2A agents

.github/workflows/docs.yml

@@ -32,5 +32,8 @@ jobs:
           git config --global user.name "github-actions[bot]"
           git config --global user.email "github-actions[bot]@users.noreply.github.com"
 
+      - name: Validate docs in strict mode
+        run: hatch run docs:build --strict
+
       - name: Deploy docs to GitHub Pages
         run: hatch run docs:deploy

.github/workflows/pypi.yml

@@ -5,6 +5,9 @@ on:
     tags:
       - "v[0-9].[0-9]+.[0-9]+*"
 
+env:
+  HATCH_VERSION: "1.16.5"
+
 jobs:
   release-on-pypi:
     runs-on: ubuntu-latest
@@ -17,7 +20,7 @@ jobs:
         uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
 
       - name: Install Hatch
-        run: pip install hatch
+        run: pip install hatch==${{ env.HATCH_VERSION }}
 
       - name: Build
         run: hatch build

.github/workflows/tests.yml

@@ -40,6 +40,18 @@ jobs:
 
   tests-haystack-v3:
     runs-on: ubuntu-latest
+    services:
+      redis:
+        image: redis:6.2-alpine
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 2s
+          --health-timeout 2s
+          --health-retries 20
+    env:
+      HAYHOOKS_TEST_REDIS_URL: redis://127.0.0.1:6379/15
     strategy:
       matrix:
         python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
@@ -55,15 +67,21 @@ jobs:
 
       - name: Install Haystack v3 into the test env
         run: |
-          hatch env create test
-          hatch -e test env run -- uv pip install --upgrade "haystack-ai>=3"
-          hatch -e test env run -- python -c "import haystack, sys; v = haystack.__version__; print('haystack', v); sys.exit(0 if int(v.split('.')[0]) >= 3 else 'expected Haystack v3+, got ' + v)"
+          hatch env create test-v3
+          hatch -e test-v3 env run -- uv pip install --upgrade "haystack-ai>=3"
+          hatch -e test-v3 env run -- python -c "import haystack, sys; v = haystack.__version__; print('haystack', v); sys.exit(0 if int(v.split('.')[0]) >= 3 else 'expected Haystack v3+, got ' + v)"
 
-      - name: Run unit tests
-        run: hatch run test:unit
+      - name: Run tests
+        run: hatch run test-v3:all
+
+      - name: Run durable process-recovery smoke test
+        if: matrix.python-version == '3.12'
+        env:
+          HAYHOOKS_TEST_PROCESS_RECOVERY: "1"
+        run: hatch run test-v3:all tests/test_durable_process_recovery.py
 
       - name: Ty - check types (Haystack v3)
-        run: hatch run test:types
+        run: hatch run test-v3:types
 
   linting:
     runs-on: ubuntu-slim
@@ -83,6 +101,9 @@ jobs:
       - name: Ty - check types
         run: hatch run test:types
 
+      - name: Build documentation in strict mode
+        run: hatch run docs:build --strict
+
   dashboard-tests:
     runs-on: ubuntu-latest
     steps:

.gitignore

@@ -192,6 +192,11 @@ chainlit.md
 .env.local
 .env.*.local
 
+# Local A2A long-running demo artifacts
+.a2a-long-running-demo-session.json
+/a2a-req
+/a2a-res
+
 # Temporary files
 *.tmp
 *.temp

README.md

@@ -7,6 +7,7 @@ With Hayhooks, you can:
 - 📦 **Deploy your Haystack pipelines and agents as REST APIs** with maximum flexibility and minimal boilerplate code.
 - 🛠️ **Expose your Haystack pipelines and agents over the MCP protocol**, making them available as tools in AI dev environments like [Cursor](https://cursor.com) or [Claude Desktop](https://claude.ai/download). Under the hood, Hayhooks runs as an [MCP Server](https://modelcontextprotocol.io/docs/concepts/architecture), exposing each pipeline and agent as an [MCP Tool](https://modelcontextprotocol.io/docs/concepts/tools).
 - 🤝 **Expose your Haystack pipelines and agents over the [A2A protocol](https://a2a-protocol.org)** (`pip install "hayhooks[a2a]"`), so other agents can discover them through auto-generated agent cards and delegate tasks to them via `hayhooks a2a run`.
+- ♻️ **Run Haystack 3 pipelines and agents as durable background work** (`pip install "hayhooks[durable]"`) with Redis-backed checkpoints, retries, cancellation, wait/resume, progress, and process recovery.
 - 💬 **Integrate your Haystack pipelines and agents with [Open WebUI](https://openwebui.com)** as OpenAI-compatible chat completion backends with streaming support.
 - 🖥️ **Embed a [Chainlit](https://chainlit.io/) chat UI** directly in Hayhooks with `pip install "hayhooks[chainlit]"` and `hayhooks run --with-chainlit` -- zero-configuration frontend with streaming, pipeline selection, and custom UI widgets.
 - 🕹️ **Control Hayhooks core API endpoints through chat** - deploy, undeploy, list, or run Haystack pipelines and agents by chatting with [Claude Desktop](https://claude.ai/download), [Cursor](https://cursor.com), or any other MCP client.
@@ -53,6 +54,7 @@ from hayhooks import BasePipelineWrapper, async_streaming_generator
 def weather_function(location):
     return f"The weather in {location} is sunny."
 
+
 weather_tool = Tool(
     name="weather_tool",
     description="Provides weather information for a given location.",
@@ -64,6 +66,7 @@ weather_tool = Tool(
     function=weather_function,
 )
 
+
 class PipelineWrapper(BasePipelineWrapper):
     def setup(self) -> None:
         self.agent = Agent(
@@ -73,7 +76,7 @@ class PipelineWrapper(BasePipelineWrapper):
         )
 
     # This will create a POST /my_agent/run endpoint
-    # `question` will be the input argument and will be auto-validated by a Pydantic model
+    # `question` will be the input argument and will be auto-validated by a Pydantic model
     async def run_api_async(self, question: str) -> str:
         result = await self.agent.run_async(messages=[ChatMessage.from_user(question)])
         return result["last_message"].text
@@ -82,9 +85,7 @@ class PipelineWrapper(BasePipelineWrapper):
     async def run_chat_completion_async(
         self, model: str, messages: list[dict], body: dict
     ) -> AsyncGenerator[str, None]:
-        chat_messages = [
-            ChatMessage.from_openai_dict_format(message) for message in messages
-        ]
+        chat_messages = [ChatMessage.from_openai_dict_format(message) for message in messages]
 
         return async_streaming_generator(
             pipeline=self.agent,
@@ -154,11 +155,22 @@ Or chat with it in the [embedded Chainlit UI](docs/features/chainlit-integration
 - Built-in support for handling file uploads in pipelines
 - Perfect for RAG systems and document processing
 
+### ♻️ Durable Execution
+
+- Run typed Pipeline and Agent work outside the request and recover it after a process restart
+- Preserve checkpoints, bounded retries, progress, cancellation, wait/resume, idempotency, and owner isolation
+- Project managed durable Agents over A2A and retain terminal results with Redis TTL
+
+Durable execution is intentionally an at-least-once engine for low-to-moderate workloads with one to three replicas.
+It uses fixed polling and cooperative cancellation; it is not a DAG orchestrator, high-scale fair queue, live migration
+system, or exactly-once boundary for external side effects. See the [supported scope and tradeoffs](docs/advanced/durable-engine.md#supported-scope-and-tradeoffs).
+
 ## Next Steps
 
 - [Quick Start Guide](docs/getting-started/quick-start.md) - Get started with Hayhooks
 - [Installation](docs/getting-started/installation.md) - Install Hayhooks and dependencies
 - [Configuration](docs/getting-started/configuration.md) - Configure Hayhooks for your needs
+- [Durable Engine](docs/advanced/durable-engine.md) - Understand restart-safe execution and its boundaries
 - [Tracing Dashboard Frontend](dashboard/README.md) - Local dashboard setup and frontend development commands
 - [Examples](docs/examples/overview.md) - Explore example implementations
 

docs/advanced/durable-engine-vs-temporal.md

@@ -0,0 +1,67 @@
+# Hayhooks durable engine and Temporal
+
+Hayhooks provides focused durable execution for Haystack 3 Pipelines and Agents. Temporal is a general-purpose
+durable workflow platform.
+
+## Core comparison
+
+| Requirement | Hayhooks durable engine | Temporal |
+|---|---|---|
+| Persistence and recovery | Redis records plus explicit Pipeline or Agent checkpoints | Event History plus deterministic [Workflow replay](https://docs.temporal.io/workflows) |
+| Delivery safety | At-least-once with fenced leases and one active owner | Workflow logic is effectively once; [Activities](https://docs.temporal.io/activity-execution) may retry |
+| Retries | Explicit, bounded retries from the latest checkpoint | Declarative, independently configurable [Activity retry policies](https://docs.temporal.io/encyclopedia/retry-policies) |
+| Interaction | Typed inspect, wait/resume, progress, and result APIs | [Queries, Signals, and Updates](https://docs.temporal.io/encyclopedia/workflow-message-passing) plus durable Workflow state |
+| Cancellation | Cooperative checks at safe boundaries | Cooperative cancellation with propagation policies |
+| Orchestration | One Pipeline or Agent execution with delayed retries | Durable timers, Activities, [schedules](https://docs.temporal.io/schedule), and [Child Workflows](https://docs.temporal.io/child-workflows) |
+| Versioning | Exact revision gate prevents incompatible recovery | Replay-safe patching and [Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning) |
+| Haystack example | Run a RAG Pipeline with `checkpoint_at=["generator"]`; after a crash, restore its `PipelineSnapshot` before generation | Invoke retrieval and generation as separate Activities wrapping Haystack components; completed Activity results are not repeated during Workflow replay |
+| Best fit | Focused, moderate-scale durable Haystack workloads | Large-scale or cross-service orchestration around Haystack |
+
+## Hayhooks code map
+
+| Concern | Relevant implementation |
+|---|---|
+| Lifecycle and fencing | [`engine.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/engine.py) |
+| Store contract and Redis persistence | [`store.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/store.py), [`redis.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/redis.py) |
+| Pipeline and Agent checkpoints | [`adapters.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/adapters.py) |
+| Retries and worker recovery | [`context.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/context.py), [`manager.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/manager.py) |
+| Progress, wait/resume, and inspection | [`context.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/context.py), [`routes.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/server/durable/routes.py) |
+| Cooperative cancellation | [`context.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/context.py), [`engine.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/engine.py) |
+| Revision and deployment safety | [`runtime.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/durable/runtime.py), [`deploy_utils.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/server/utils/deploy_utils.py) |
+| Durable A2A projection and recovery | [`durable_executor.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/server/a2a/durable_executor.py), [`redis_task_store.py`](https://github.com/deepset-ai/hayhooks/blob/main/src/hayhooks/server/a2a/redis_task_store.py) |
+
+Hayhooks makes an existing Haystack Pipeline or Agent durable with minimal restructuring. Temporal offers finer-grained
+orchestration, but obtaining that granularity usually means deciding which Haystack operations should become separate
+Activities; the whole Pipeline can still run as one Activity when independent step recovery is unnecessary.
+
+## What Temporal adds
+
+| Gain | Why it matters |
+|---|---|
+| Independent step policies | Retrieval, generation, payments, or notifications can have separate retries, timeouts, workers, and resource limits |
+| Cross-service orchestration | A Workflow can coordinate Haystack with databases, external APIs, approval systems, and other services |
+| Durable time and interaction | Native timers, schedules, callbacks, Signals, Queries, and Updates |
+| Horizontal routing | Activities can run on different task queues, worker pools, languages, or infrastructure |
+| Production operations | Execution history, search, UI, batch operations, metrics, and mature failure investigation |
+| Long-running deployments | Worker versioning, pinning, gradual rollout, rollback, and Workflows that span releases |
+| Reduced engine ownership | Temporal owns task delivery, persistence, recovery, and orchestration semantics instead of Hayhooks maintaining them in Redis |
+
+For restart-safe Haystack Pipelines and Agents with checkpoints, retries, cancellation, and resume, Temporal provides
+little immediate functional gain and adds infrastructure plus integration work. It becomes valuable when an execution
+grows into a long-lived, multi-step workflow spanning Haystack and other systems.
+
+Temporal does not remove the need for idempotent external side effects: an Activity may execute more than once when its
+result is lost and the task is retried.
+
+## References
+
+- [Hayhooks durable engine](durable-engine.md)
+- [Temporal Workflows](https://docs.temporal.io/workflows)
+- [Temporal Activities](https://docs.temporal.io/activities)
+- [Temporal Activity execution](https://docs.temporal.io/activity-execution)
+- [Temporal retry policies](https://docs.temporal.io/encyclopedia/retry-policies)
+- [Temporal Workflow message passing](https://docs.temporal.io/encyclopedia/workflow-message-passing)
+- [Temporal Child Workflows](https://docs.temporal.io/child-workflows)
+- [Temporal Schedules](https://docs.temporal.io/schedule)
+- [Temporal Visibility](https://docs.temporal.io/visibility)
+- [Temporal Worker Versioning](https://docs.temporal.io/production-deployment/worker-deployments/worker-versioning)