Workflow authoring skill: any agent can author, validate, and run a Python workflow script

[fanhongy]

Goal

Any CAO agent (developer, supervisor, reviewer — doesn't matter) can take a user's natural-language request ("run a data analysis over these files"), author a Python workflow script for it, validate it, and — after explicit user approval — run it against cao-server.

Follow-on to #312 / PR #399 (script tier). This is the authoring-skill piece: today the only workflow skill teaches YAML, so agents produce the wrong artifact.

User flow

1. User talks to any agent in a CAO session.
2. User asks for something workflow-shaped (e.g. multi-file data analysis).
3. Agent authors the .py workflow script via the skill — or, if it lacks write permission, hands off authoring to a developer agent, passing the skill name.
4. Agent validates the script (cao workflow validate) and fixes findings.
5. Agent presents the plan and asks the user whether to run it. Never auto-runs.
6. On approval, agent invokes the run with an explicit, pre-announced run id and reports the status/cancel handles.

Design decisions

• Rewrite workflow-author in place (single authoring path, Python-first; YAML demoted to describe/display). Skill is discovered via the existing skill catalog + load_skill, so all agent profiles inherit it with zero per-profile wiring.
• Validate gate is mandatory between author and ask-to-run. The .py lint findings (fan-out without step_id, disallowed imports, nondeterminism) are load-bearing, not advisory.
• Explicit --run-id, announced before the run starts — until a cao workflow runs list verb exists, an unnamed long run is uncancellable mid-flight.
• Blocking vs background: short runs use the synchronous workflow_run MCP tool; long runs are backgrounded via bash (cao workflow run <name> --run-id <id> --json &) so the agent's terminal stays free.
• Hard rule: no path skips user approval before execution (the script tier executes generated Python).

Operational facts the skill must teach

• Script lives at ~/.aws/cli-agent-orchestrator/workflows/<name>.py; run by stem (no path separators; no same-stem .yaml sibling).
• Fan-out: ThreadPoolExecutor, max_workers 3–5, explicit stable step_id per concurrent call, sorted() inputs (determinism obligation — resume relies on it).
• Big outputs: agents write files and return paths; per-step output is null for schema-less steps, so emit_output / files are the source of truth.
• Failure recovery: cao workflow resume <run-id> replays completed steps from the journal.
• Provider/agent guidance: use a headless provider (claude_code); kiro_cli has an open TUI bug.

Acceptance criteria

• From a fresh session, an agent asked to "create a workflow for X" produces a validated .py script (not YAML) via the skill.
• A read-only agent hands off authoring to developer correctly.
• The agent asks before running; on approval the run starts with a pre-announced run id; the user is told how to check status and cancel.
• A fan-out script authored via the skill passes validate with explicit step_ids and survives a resume after a killed run.

Dependencies / relations

• PR feat(workflow): script-tier run surface + author shim — U5+U6 (#312) #399 (script tier run surface) — merged or near-merge prerequisite.
• Relates to the run-discoverability gap (cao workflow runs, id-at-submit) — separate scope, but the skill's run-id rule is the interim mitigation.
• [Bug] workflow run-step env_vars silently dropped with session_name — CAO_WORKFLOW_RUN_ID lost, workflow_return misroutes #408 / [Bug] workflow run sticks in 'running' (step waits completed, provider settles idle) and cancel never interrupts the wait #409 fixes (env_vars on session_name; idle completion + interruptible cancel) already validated live with claude_code agents.

[fanhongy]

Addendum from a design session distilling the one script-tier workflow we've actually built end-to-end (the tiered PR-review script). The issue's lifecycle, validate-gate, no-auto-run, run-id, background-vs-blocking, and resume guidance all stand — below is only what's missing or needs reconciling, so I'm not repeating what's already here.

Add: workflows as parameterized tools (the input-delivery gap)

The Goal ("take a NL request → author a script for it → run it") today implies authoring a bespoke script per request, because the script tier can't take runtime inputs. The runner receives and journals inputs, but never delivers them to the subprocess — the spawn env is a fixed 5-key allowlist. So parameterization degenerates to one-file-per-run with an in-source constant.

The stronger model, and what makes "any agent runs a workflow from NL" actually tool-like: author once, invoke with NL-extracted inputs. The agent reads a workflow's declared input schema, pulls args from the user's request, and calls it — exactly like calling a tool. Five small pieces:

1. Declare — module-level INPUTS literal in the script, mirroring YAML's inputs: vocab (string|int|bool|path), AST-extractable without executing. This literal is the tool's input schema.
2. Validate — engine checks passed inputs against it (types/required/defaults), reusing the YAML tier's validator.
3. Deliver — add ONE key, CAO_WORKFLOW_INPUTS (validated JSON), to the constructed spawn env. Still a constructed allowlist — one engine-generated key, like CAO_WORKFLOW_RUN_ID; no ambient-env leak.
4. Read — shim gains get_inputs() -> dict.
5. Resume-safe — resume re-delivers the journaled inputs verbatim, so input-driven control flow stays deterministic.

This is an engine+shim change (sibling scope, not the skill itself), but it's a prerequisite for the skill's headline capability — without it the skill can only teach the constant-editing hack. The skill must teach the INPUTS/get_inputs() pattern once delivery lands.

Invocation surface for the MVP: cao-mcp-server's workflow_run (mandatory, always present) + whatever MCP servers the user has installed/trusted. Auto-exposing each indexed workflow as its own dynamic MCP tool is a deferred follow-on, gated to user-trusted servers with cao-mcp-server required.

Add: secrets discipline (skill-owned, no engine flag)

Inputs are journaled in plaintext and replayed on resume. Rather than an engine secret: true flag (which would create an un-replayable-input special case), the skill teaches the agent to recognize secret-shaped values and refuse to pass them as inputs — pass a reference/name and resolve from a secrets manager / env at step time.

Add: runtime invariants from the PR-review build the skill should teach

The issue's operational facts miss the lessons that actually cost us runs:

• Match step to role capability. This qualifies the "agents write files and return paths" guidance: only write-capable roles (developer: fs_*+execute_bash) may write. A read-only role (reviewer) told to write hangs on a permission prompt for the full step budget and fails the run — read-only steps must READ inputs and RETURN findings inline. (This is distinct from the authoring-permission handoff already in the issue; it's about the steps the script drives.)
• Fault-tolerance per concurrent unit. Catch ShimError inside the thread body so one timed-out step degrades to "unavailable" and synthesis proceeds with survivors, instead of one 504 sinking the whole run. Optional contention-free solo-retry of a failed unit.
• Two timeout ceilings — the concrete reason to background long runs: per-step timeout is unbounded, but the whole-script ceiling (~8700s) AND the MCP client idle timeout (~1800s) both bite. A giant-diff run that blows 1800s loses its return value (and the run id, until the discoverability work lands). This is why the run-id-at-submit + background rule matters, not just that it does.
• Lint-gate rationale (worth stating so authors treat findings as load-bearing): the import cli_agent_orchestrator ban enforces the HTTP-only boundary (script talks to the server only via the shim); the random/time/datetime/uuid nondeterminism warnings exist because resume re-executes the script from the top and replays journaled results — wall-clock/random control flow → ReplayDivergenceError.

Add: YAML-as-projection, ranked

"YAML demoted to describe/display" — worth pinning which projection. For a progress UI/TUI the runtime journal is the primary source (it reflects what actually ran, including dynamically-generated steps a loop/runtime fan-out produces). A static script→YAML preview (AST-extract run_step calls) is an optional, inherently lossy before-run nicety — never the truth source, never an authoring target.

Reconcile (conflicts with the current issue text)

1. Fan-out width. The issue says max_workers 3–5. Measured on the PR-review script, 4 concurrent claude_code reviewers reliably starved the heaviest lens (600s timeout on nearly every run); dropping to 2 fixed it. This is host/provider-dependent, but 3–5 will likely reproduce the starvation for claude_code. Suggest a conservative default (2 for claude_code) exposed as a tunable, with the tradeoff documented — rather than 3–5 as the taught value.
2. Skill packaging. The issue says "rewrite workflow-author in place." Today we leaned toward a new CAO-level skill (the load_skill / cao-* tier — provider-agnostic, same category as cao-memory), retiring workflow-author once it lands. Same spirit (one Python-first path, all agents), different mechanics — worth picking one explicitly.