feat: add DBOS integration for durable Agent execution

[julian-risch]

Related Issues

• Proposal: deepset-ai/haystack-private#542

Proposed Changes:

Adds dbos-haystack, a new integration that makes Haystack Agent runs durable with DBOS — an MIT-licensed, in-process library backed by Postgres or SQLite, with no external orchestrator to deploy.

• DBOSChatGenerator wraps any Chat Generator and runs each model call inside a @DBOS.step(). The step records the replies as ChatMessage.to_dict() rather than objects, so the checkpoint stays readable, survives DBOS's portable-JSON serializer, and doesn't unpickle into a changed dataclass if haystack-ai is upgraded between the crash and the recovery. Outside a workflow the wrapper is transparent, so the same object works in tests and in applications that never launch DBOS.
• durable_agent() returns a copy of an Agent with its generator wrapped, and warns when the Agent carries a toolset that discovers its tools while running.
• DBOSConfirmationStrategy plugs into the stock ConfirmationHook with no new hook class. It publishes the pending tool call as a DBOS event and waits on DBOS.recv, so a run can suspend for hours or days awaiting approval and survive a restart while it waits.

The user writes their own @DBOS.workflow around agent.run(). There is deliberately no DBOSAgent wrapper that owns the decorator: DBOS persists workflow inputs, so promoting Agent.run to a workflow would require pickling messages, streaming_callback and runtime tools — a plain callback or a lambda-backed Tool makes the workflow fail to start, with the error surfacing deep inside DBOS. Pydantic AI hit exactly this and deprecated its own DBOSAgent in favour of this shape.

Recovery in DBOS is replay, not forward-resume: the workflow body re-executes and completed steps return their recorded output. Tool calls are not checkpointed in this version and therefore run again, so tools must be idempotent or be steps themselves. That is documented in a durability-guarantees table in the README, together with the other consequences (streaming does not replay, tracing spans are re-created, dynamic toolsets are unsupported, sync and async must not be mixed). Auto-wrapping tools is deliberately left out for now — see "Notes for the reviewer".

Also included: a README with a quickstart and the caveats, a runnable examples/durable_agent.py that crashes and then recovers, pydoc config, and the scaffold-generated CI workflow, labeler entry and README inventory row.

How did you test it?

From integrations/dbos: hatch run fmt-check, hatch run test:types, hatch run test:unit (54 passed) and hatch run docs all pass locally.

The whole suite runs against in-process SQLite, so no service container and no secrets are needed — the workflow keeps the "no integration tests yet" comment in place of the integration-test steps.

Replay is tested two ways, both taking the same path a crash-restart would, and neither needing a subprocess or a sleep:

• fork_workflow restarts a failed workflow from a chosen step.
• cancel_workflow → resume_workflow picks up a workflow interrupted mid-flight.

Both assert the fake generator was called exactly once across the original and recovered runs. A companion test asserts an unwrapped tool's counter reaches 2, so the at-least-once contract is pinned as tested behaviour rather than only a documentation claim. The human-in-the-loop suite covers confirm, reject, modify, timeout, and — the decisive one — that a recovered run reuses the recorded decision and executes the tool without a second DBOS.send.

Not covered by automated tests: a real kill -9 against a live process, and Postgres as the system database. examples/durable_agent.py exercises the former by hand against a real OpenAIChatGenerator.

Notes for the reviewer

Opening as a draft — the proposal issue is still open, and two decisions are worth your input before this lands:

1. Agent.clone is not in haystack-ai==3.0.0, so durable_agent rebuilds the Agent from its init signature. That surfaced a second divergence: on 3.0.0, Agent.state_schema holds the resolved schema including reserved keys, which cannot be passed back to __init__; the value that was passed in lives in _state_schema. Handled by a documented attribute map in durability.py, which keeps working once clone ships. Alternatively we wait for that release and raise the haystack-ai floor.
2. Tool-step wrapping is deferred, not forgotten. Haystack's synchronous tool path always submits to a ThreadPoolExecutor with a copied context, and DBOS assigns step ids by mutating a counter on a single shared context object. With tool_concurrency_limit > 1 two @DBOS.step tools can be handed the same id, which either fails recovery loudly or — when two calls to the same tool collide — returns one call's cached result for the other. The async path is safe, since tool coroutines are created in call order and the semaphore wakes waiters FIFO. Shipping an auto-wrapper needs that resolved first; the README documents the manual @DBOS.step escape hatch and its constraints in the meantime.

Related, though solving the problem from the other direction: deepset-ai/hayhooks#253 adds a Redis-backed durable engine inside the Hayhooks server, checkpointing Agent State through the hook seams and resuming forward. This integration is the library-shaped counterpart — plain Python, Postgres or SQLite, replay-based, no engine to operate. Worth a view on whether the READMEs should cross-reference each other once that lands.

Suggested review order: chat_generator.py for the step boundary, durability.py for the rebuild and guard rails, human_in_the_loop.py for the durable wait, then tests/test_recovery.py.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes — not yet; the design changes above should go on deepset-ai/haystack-private#542 once the two open questions are settled
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

🤖 Generated with Claude Code

[github-actions]

Coverage report (dbos)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
integrations/dbos/src/haystack_integrations/dbos
_runtime.py					46
chat_generator.py					135, 207-208, 221-222
durability.py					29-30, 42
human_in_the_loop.py					72, 209, 270-274
Project Total

This report was generated by python-coverage-comment-action

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​dbos@​2.29.0

View full report