Skip to content

feat(engine): resolve registry references in sub-workflow workflow: field#188

Merged
jrob5756 merged 4 commits into
mainfrom
issue/172-registry-subworkflow-refs
May 14, 2026
Merged

feat(engine): resolve registry references in sub-workflow workflow: field#188
jrob5756 merged 4 commits into
mainfrom
issue/172-registry-subworkflow-refs

Conversation

@jrob5756
Copy link
Copy Markdown
Collaborator

Closes #172.

What

Extends type: workflow agents to accept registry references (name@registry#ref syntax) in the workflow: field, in addition to local file paths. Connects the existing sub-workflow composition feature (#79) and the workflow registry, enabling cross-team workflow composition by reference.

agents:
  - name: analyze
    type: workflow
    workflow: 'analysis@team-a#v2.0.0'    # NEW — was local path only
    input_mapping:
      repo: "{{ workflow.input.repo }}"

How

Engine (src/conductor/engine/workflow.py)

New _resolve_subworkflow_path() async helper, used by both _execute_subworkflow and _execute_subworkflow_with_inputs. Resolution order:

  1. If the value resolves to an existing file relative to the parent workflow directory → return that path. Preserves backward compatibility for bare names like analysis referring to a sibling file without a .yaml extension.
  2. Otherwise, parse via registry/resolver.resolve_ref() to detect file vs registry syntax.
  3. File-kind that doesn't exist → return the path so the caller emits a clear "file not found" error.
  4. Registry-kind → call cache.fetch_workflow() via asyncio.to_thread() (synchronous HTTP) and return the local cached path.

RegistryError from both resolve_ref() and fetch_workflow() is wrapped in ExecutionError with agent context.

Validator (src/conductor/config/validator.py)

conductor validate now resolves and recursively validates sub-workflow references for all type: workflow agents (including for_each inline agents). For registry refs, this fetches the workflow into the local cache during validation. This is a behavioral change: validate now makes network requests when registry refs are present (the issue calls for this explicitly to validate the full composition tree).

Validation is recursive with cycle detection (visited-set threaded through validate_workflow_config via internal kwargs) and a depth cap of 10.

Resume / replay compatibility

The resolver is called on every sub-workflow execution, including after conductor resume. For SHA-pinned refs (name@registry#v1.2.3 or name@registry#<sha>) the cache is content-addressable so resume is fully deterministic. For mutable refs (#main or omitted) resume may pick up a newer commit if the upstream branch has moved — documented in the helper's docstring with a recommendation to pin in production.

Tests

13 new tests covering:

  • Engine: registry ref resolves and executes (mocked); registry fetch failure → ExecutionError; extensionless local file takes precedence over registry; malformed ref → ExecutionError; full round-trip resume verifying the same cached path is used on both runs.
  • Validator: local path OK; missing local errors; malformed ref errors; registry ref fetch + recursive validate; registry fetch failure errors; for_each inline workflow agent validated; A → B → A cycle detection.

All 2445 tests pass; make check (ruff + ty) clean.

Files

src/conductor/config/validator.py     | +178
src/conductor/engine/workflow.py      | +108
tests/test_config/test_validator.py   | +312 (TestSubWorkflowRefValidation)
tests/test_engine/test_subworkflow.py | +370 (TestRegistrySubWorkflowResolution)

Design note

The registry design doc states "Keep the registry mechanism out of the engine: it is a CLI/loader concern." This PR crosses that line by adding registry resolution to the engine's sub-workflow path-resolution step. Per the issue's analysis, the alternative (pre-fetch and reference by local path) defeats the purpose of composable workflows. The change is contained to a single helper called immediately before load_config().

jrob5756 and others added 4 commits May 14, 2026 09:30
…ield

Extends type: workflow agents to accept registry references
(workflow[@registry][#ref] syntax) in the workflow: field, not
just local file paths. Resolves issue #172.

Resolution order in _resolve_subworkflow_path():
1. If agent_workflow resolves to an existing file relative to the
   parent workflow directory, return that path (backward compat —
   preserves extensionless local file references like 'analysis').
2. Parse via registry/resolver.resolve_ref() to detect file vs
   registry syntax.
3. If file-kind, return the (non-existent) path so the caller emits
   a clear "file not found" error.
4. If registry-kind, call cache.fetch_workflow() via asyncio.to_thread()
   to fetch and cache the workflow, returning the local cached path.

Both _execute_subworkflow and _execute_subworkflow_with_inputs use the
new helper. RegistryError from resolve_ref() and fetch_workflow() are
wrapped in ExecutionError with agent context.

conductor validate now resolves registry refs for all type: workflow
agents (including for_each inline agents) when workflow_path is
provided. Validation is recursive with cycle detection and a depth
cap of 10.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion

The previous implementation built a `_visited` set in
`_validate_subworkflow_refs` but called `validate_workflow_config`
recursively without threading the set through. Each recursive level
started with a fresh empty `_visited`, so cycles like A → B → A were
not detected and would either recurse to the depth cap silently or
loop until an unrelated failure (e.g. fetch error).

Adds two internal-only kwargs to `validate_workflow_config`
(`_visited_subworkflows`, `_subworkflow_depth`) that are threaded
through recursive validation so the visited set accumulates across
the full sub-workflow chain. Adds a test exercising A → B → A.

Also adds a docstring note to `_resolve_subworkflow_path` explaining
that mutable registry refs (e.g. `name@registry#main`, or no `#ref`)
may resolve to a different commit on `conductor resume` if the
upstream branch has moved. Pinned tags or commit SHAs guarantee
deterministic resume.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a full round-trip test for the documented resume + registry ref
behavior: fail mid-sub-workflow → checkpoint → resume → success. The
test verifies that:

1. fetch_workflow is called again on resume (resolution is not
   memoized on the engine instance), and
2. for a stable cached ref, the resolved path is identical between
   the original run and the resumed run.

This is the determinism guarantee for cached/SHA-pinned registry
sub-workflow refs across resume.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Code-review findings:

* **Cycle detection now uses inode identity** (`(st_dev, st_ino)`)
  instead of `str(Path.resolve())`. The previous string-based approach
  still caught cycles eventually but treated case-variants of the same
  file as distinct entries on case-insensitive filesystems (macOS,
  Windows). Inode-based identity catches the cycle on the first
  revisit and correctly handles symlinks too. Adds a test that
  exercises the case-variant scenario (skipped on case-sensitive FS).

* **Depth-limit silent return now emits a warning.** Previously
  `_validate_subworkflow_refs` returned empty errors/warnings when
  `_MAX_SUBWORKFLOW_VALIDATION_DEPTH` was hit, so the user couldn't
  tell whether the workflow was clean or whether validation was
  truncated. Adds a regression test that builds a chain longer than
  the limit and asserts the warning surfaces.

* **Bare `except Exception` narrowed to `except ConfigurationError`**
  for the recursive `validate_workflow_config` call (the only
  exception that recursive call documents raising). The `load_config`
  catch stays broad — load can fail many ways (YAML parse errors,
  schema errors, file IO).

Test-quality findings:

* `test_registry_ref_resolved_and_executed` no longer mocks
  `_resolve_subworkflow_path` — it now mocks only the registry config
  loader and `fetch_workflow`, so the engine's real resolution flow
  runs end-to-end (real `resolve_ref` parses `analysis@team-a#v1.0.0`,
  real precedence check confirms no local file shadows it, real
  registry branch is taken).

* `test_registry_ref_validates_fetched_workflow` no longer mocks
  `resolve_ref` — it captures `fetch_workflow` arguments and asserts
  the parsed workflow name, registry name, and ref are correct,
  exercising real parser logic.

* `test_local_file_takes_precedence_over_registry` now patches
  `resolve_ref` and asserts it is not called when a local file exists,
  guaranteeing the precedence short-circuit is preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jrob5756 jrob5756 merged commit efd3f98 into main May 14, 2026
9 checks passed
@jrob5756 jrob5756 deleted the issue/172-registry-subworkflow-refs branch May 14, 2026 17:08
@jrob5756 jrob5756 mentioned this pull request May 14, 2026
jrob5756 added a commit that referenced this pull request May 14, 2026
- feat(engine): registry references in sub-workflow workflow: field (#188)
- feat: Claude Code plugin marketplace (#186)
- docs(skill): refresh conductor skill with latest features (#187)
- docs: rewrite 'Why Conductor?' README section (#185)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: Registry-resolved sub-workflow references

1 participant