docs: Pipeline flexibility — architectural notes and design decisions

[saraichinwag]

Context

From a pipeline flexibility audit tracing through Engine.php → StepNavigator → AIStep → ToolExecutor → PublishStep → AgentPingStep. These are architectural observations, not bugs — documenting them so they're visible when designing new pipeline patterns.

Current Architecture (v0.26.0)

How It Works

• Steps execute sequentially via execution_order (integer, 0-indexed)
• StepNavigator finds next step by current_order + 1 — strictly linear
• AI steps see handler tools from adjacent steps only (previous + next via ToolExecutor::getAvailableTools)
• Engine data accumulates across steps via datamachine_merge_engine_data()
• Any step type can appear at any position, multiple times, no constraints

Design Decisions (Intentional, Not Bugs)

1. Linear execution only — no branching
Steps run in order. No "if step succeeds → go to step 3, else → step 5." Failures stop the pipeline (or complete with status override). This keeps the execution model simple and predictable.

When this matters: If you want "publish to Pinterest only if article has an image" — the AI step needs to make that judgment and either call or skip the Pinterest tool. The pipeline itself doesn't branch.

2. AI tool visibility is ±1 step
An AI step only sees handler tools from its immediate predecessor and successor. This is by design — it scopes what the AI can do at each point in the pipeline.

When this matters: Fetch → AI → WP Publish → Pinterest Publish won't work because the single AI step only sees WP Publish tools (next step), not Pinterest. Solution: add a second AI step between WP and Pinterest. Each publish handler needs its own preceding AI step.

3. Publish steps require AI tool results
PublishStep::executeStep() uses ToolResultFinder::findHandlerResult() to find matching tool call results in data packets. If the preceding AI didn't call the handler's tool, the Publish step returns empty and the job fails.

When this matters: Every Publish step needs an AI step before it that actually calls the tool. There's no "auto-publish" mode where the Publish step acts on engine_data alone without an AI decision.

4. Agent Ping is fire-and-forget
Agent Ping sends the webhook and continues to the next step immediately. It does NOT wait for a response. For mid-pipeline approval gates, use WebhookGate instead (parks the pipeline until callback).

5. Single data packet stream
All steps share one dataPackets array. Each step appends to it. Later steps see everything from earlier steps. The AI needs clear prompting to know which results to act on.

Potential Future Enhancements (Not Currently Needed)

These are patterns that might be useful someday. Filing here for reference, not as feature requests.

• Optional/skippable steps — Already handled: AI can call skip_item tool to set job_status override, and the engine completes the pipeline gracefully with agent_skipped status instead of failing. Not a gap.
• Conditional routing — Branch to different steps based on step results or engine_data values
• Extended tool visibility — Option to expose handler tools from non-adjacent steps
• Auto-publish mode — Publish step that acts on engine_data directly without requiring an AI tool call (for deterministic publishing where AI judgment isn't needed)

Recommended Pipeline Patterns

Content + Pinterest

Fetch → AI (content) → WP Publish → AI (pin copy) → Pinterest Publish

Content + Review Gate + Pinterest

Fetch → AI (content) → WP Publish → Agent Ping (notify) → WebhookGate (approval) → AI (pin copy) → Pinterest Publish

Content + Multi-Platform

Fetch → AI (content) → WP Publish → AI (social copy) → Pinterest Publish → AI (tweet copy) → Twitter Publish

Each social platform needs its own AI step for platform-optimized copy.