[open-actions-fake-strategist] Support structured matrix values in workflow strategies #168

🤖 Open Actions Strategist Agent @gjkim42

Latest verdict

• Status: VALID
• Checked at: 2026-11-03T00:20:38Z
• Evidence: The default branch moved from dcdfdf2 to e9134c6. The new commits are #201 (v1 cache protocol) and #202 (paths/paths-ignore, part of #116). Neither touches matrix code: git diff dcdfdf2 e9134c6 changes no line mentioning a matrix in internal/workflow/workflow.go, and nothing under internal/controller, internal/runner, or api.
  • The failures still reproduce on e9134c6. A temporary probe ran through make test TEST_PACKAGES=./internal/workflow TEST_FLAGS="-count=1 -v -run ZZProbe" and was then deleted; the working tree is clean.
    • GitHub's official os × node example fails with job "test" matrix axis "node" values must be scalars.
    • An axis of arrays fails with job "fuzz" matrix axis "fuzz_test" values must be scalars.
    • An object axis fails with job "test" matrix axis "config" values must be scalars.
    • An include entry with an object value fails with job "test" matrix include entry 1 values must be scalars.
    • EvaluateMatrix on a fromJSON matrix producing {"target": [{"triple": ..., "os": ...}]} fails with job "build" matrix axis "target" values must be scalars.
    • include: [{OS: a, extra: "yes"}] with os: [a, b] still returns [{OS: a, extra: yes, os: a}, {OS: a, extra: yes, os: b}], so the entry is matched case-sensitively and added to both combinations.
  • The code is unchanged; only line numbers moved. validateMatrixValue, validateMatrixEntries, resolveMatrixValue, and matrixValuesEqual moved down by five lines. WorkflowJob.spec.matrix.values is still map[string]string. matrixStringValues/matrixDisplayName and rerunMatrixValuesMatch still use fmt.Sprint. PlanVersion is still 10, and the runner still exposes "matrix": plan.Matrix. The 87db1d2 permalinks below still show the current matrix code.
  • The official docs are unchanged. In github/docs, the latest commits are still workflow-syntax.md at c67a936, contexts.md and data/reusables/actions/workflows/matrix-keys-are-case-insensitive.md at 03d2e24, and expressions.md at b32e08f.
  • The corpus was rescanned after Support workflow path filters for push and pull request events #202. This candidate still ranks first among untracked gaps, but only narrowly. See the 2026-11-03 rescan.
    • Structured matrix values are now the only parse-level blocker for 27 files in 13 repositories, up from 20 files in 12. Ten of the 27 use paths/paths-ignore, which Support workflow path filters for push and pull request events #202 unblocked.
    • Step timeout-minutes, this slot's earlier candidate, rose to 19 files in 13 repositories. It now leads on directly triggered repositories, 12 to 9.
    • The two are tied on repositories, with no clear margin either way, so the candidate is kept.
    • Step background/wait/wait-all/cancel/parallel, generally available since 2026-06-25, appear in 23 files across 7 repositories. They are the only blocker in just 1 repository.
  • There is no overlap. #65 and draft #68 have not been updated since 2026-08-12. Open PR #196 is still at head 2ecffc8 and touches no matrix code. Searches of open and closed issues and PRs for matrix find no new tracking issue or PR. This issue still has no comments or assignees.

Selected compatibility roadmap increment

Support structured values, meaning objects and arrays, in jobs.<job_id>.strategy.matrix axes, include, exclude, and expression-produced matrices. Carry them unchanged into the matrix context. Match include and exclude against them as documented.

The assessed area is matrix strategy values: how the parser, planner, controller, and runner represent matrix values and match include/exclude entries against them. The coverage pattern is a mature scalar implementation: axes, include/exclude, dynamic matrices, max-parallel, fail-fast, and reruns all work. But a single scalar-only representation runs through every layer, and it rejects a pattern the official reference documents with its own example. Rust and C++ projects use this pattern heavily for build targets and configuration matrices.

Real workloads, pinned to the commits that were scanned. The open-webui, whisper.cpp, tauri, and bat workflows are triggered directly by events. The Prometheus and protobuf workflows are reusable workflows triggered only by workflow_call, so running them unchanged also requires support for their callers (#23).

• open-webui/open-webui docker.yaml runs on push and is Linux-only. It crosses two object axes and reads them through runs-on: ${{ matrix.platform.runner }}, the step condition if: matrix.variant.free_disk, and with values.
• ggml-org/whisper.cpp docker.yml runs on schedule and is Linux-only. It uses an axis of objects in runs-on: ${{ matrix.config.runs_on }} and in with values such as matrix.config.dockerfile.
• tauri-apps/tauri docker.yml runs on workflow_dispatch and is Linux-only. It uses an axis of objects read through matrix.target.name.
• sharkdp/bat CICD.yml uses the common matrix.job: [{target, os, use-cross}] pattern in name, runs-on, step if, and with. Its Linux combinations depend only on this gap. Its Windows and macOS combinations also need matching Runners.
• prometheus/prometheus fuzzing.yml is a Linux-only reusable workflow. It uses an axis of arrays and consumes it with join(matrix.fuzz_test, ' ').
• protocolbuffers/protobuf test_cpp.yml is a reusable workflow with an object axis config and an include entry whose config value is an object (Bazel8). That entry must become a new combination. The workflow reads the values through matrix.config.runner in runs-on and matrix.config.continuous-only in if.

Prioritization evidence

On 2026-11-03, a GitHub search selected the 20 most-starred repositories for each of Go, TypeScript, Python, Rust, Java, JavaScript, and C++. Each had at least 5,000 stars, was not archived, and had been pushed since 2026-10-01. Of the 140 repositories, 133 have top-level .github/workflows/*.y{a,}ml files, 2,332 files in total, each pinned to its repository's HEAD commit at download time.

A temporary make test probe ran workflow.Parse from e9134c6 on every file, and 1,029 parse. For each failing file, the probe repeated three steps until the file parsed (1,283 files) or hit an unclassified error (20 files):

1. Classify the parser's error.
2. Remove or replace only the construct behind that gap.
3. Parse again.

A file's blockers are all the gaps it hit. "Only blocker" counts files that have exactly one blocker and parse once it is rewritten.

Each remaining rejected construct is the only blocker in at most two repositories:

• The label, discussion, delete, and status triggers.
• tags-ignore (#21).
• runs-on group mappings.
• One unsupported activity type.
• A top-level description key.

The following never appear as the only blocker:

• Concurrency-group expressions over 256 characters before evaluation (25 files).
• The branch_protection_rule trigger (12 files).
• Expressions in strategy.fail-fast/max-parallel (7 files).
• The case() expression function (7 files).
• schedule timezone keys (2 files).

The tracked gaps dominate and already have owners. Among untracked gaps, structured matrix values and step timeout-minutes tie at 13 repositories:

• Structured matrix values block more files, 27 to 19.
• Step timeout-minutes blocks more directly triggered repositories, 12 to 9, because 7 of the 27 structured-matrix files are reusable workflows that also need #23.
• Path filter support (#202) moved both up: 10 of the 27 files and 9 of the 19 use paths or paths-ignore.

Earlier scans ranked structured matrix values first among untracked gaps, at 20 files in 12 repositories, ahead of step timeout-minutes at 8 to 10 files. Those scans ran on 2026-09-30 against 0586283 (2,125 files) and on 2026-10-20 against 87db1d2 (2,274 files), using the same repository criteria with an earlier push cutoff.

These counts exclude failures that happen during job planning. A matrix built from expressions that produces objects fails only at planning, which a parse scan cannot see.

Documentation contract and current coverage

GitHub's open-source workflow parser offers supplementary implementation evidence for nested comparison. MatrixBuilder.MatrixFilter walks every nested mapping and sequence in an include/exclude entry and compares each leaf path, for example matrix.versions.node-version == 12. MatrixInclude splits entries into keys that filter original variables and extra keys.

Minimal proposal and acceptance criteria

No new CRD fields or manifest changes. The change touches only the workflow model, matrix expansion, and the controller's string form for WorkflowJob matrix values. Existing scalar matrices and existing WorkflowJobs keep their exact current representation.

1. Parsing and limits. Accept mappings and sequences at any depth in axis items and in include/exclude entry values, alongside scalars. GitHub's official array-of-objects example parses and expands to its four documented combinations. Keep the existing limits on axes, entries, and job count. Bound each structured value by the length of its canonical encoding under the existing 1,024-character per-value limit, and document the rule; the largest structured value in the 2026-11-03 corpus encodes to 457 bytes. Keep rejecting empty or non-mapping include/exclude entries. Explicitly reject axes whose names differ only by case, rather than creating two variables, as GitHub's TemplateReader does for duplicate mapping keys.

2. Expansion semantics. The Cartesian product includes structured items. include and exclude compare a structured filter value leaf by leaf, as GitHub's parser does:

   • A nested exclude filter is a partial match.
   • An include entry whose nested leaves conflict with an original value creates a new combination; protobuf's Bazel8 entry is the example.
   • Variable names compare case-insensitively: include: [{OS: a, extra: yes}] augments only the os: a combination of os: [a, b].

3. Dynamic matrices. Whole-matrix, axis, include, and exclude expressions may produce structured values, for example ${{ fromJSON(needs.setup.outputs.matrix) }} with {"target": [{"triple": "...", "os": "..."}]}. These plan successfully. Missing outputs, invalid JSON, non-array axes, and oversized results still fail with JobPlanningFailed, as documented.

4. Context propagation. Each expanded job's matrix context keeps the typed structure:

   • In controller planning expressions: job name, runs-on, timeout-minutes, continue-on-error, concurrency, and if.
   • In runner step expressions such as matrix.job.use-cross in a step if, matrix.target.name in with, join(matrix.fuzz_test, ' '), and toJSON(matrix).

5. Resource identity and reruns. Keep spec.matrix.values as map[string]string. Encode structured values in one deterministic canonical JSON form that does not depend on source key order. Use that same form for default display names (never Go map[...] formatting), matrix identity, and rerunMatrixValuesMatch, so a structured combination can be rerun from the Console. Update the field's API comment and regenerate the CRD. No job-plan version change is needed: runners that accept plan version 9 or later (the current PlanVersion is 10) already decode matrix as map[string]any and evaluate nested properties (see Assessment validation). Add a runner test that pins this behavior.

6. Conformance coverage and docs. Add tests for:

   • The official example.
   • Axes of arrays (Prometheus) and of objects (tauri, bat, protobuf).
   • Structured include entries that augment versus create combinations.
   • Nested partial exclude.
   • Case-insensitive include/exclude keys.
   • Structured dynamic matrices.
   • WorkflowJob values and display names.
   • Rerunning a structured combination.
   • Runner step if, with, and run reading nested properties.

   Update the strategy, context, and validation-limit sections of docs/reference.md to describe the implemented behavior. Verify with make update, make verify, and make test.

Out of scope:

• runs-on expressions that evaluate to arrays. ComfyUI's runs-on: ${{ matrix.runner_label }} relies on this; GitHub's parser accepts sequences, but the runs-on reference documents only "a single variable containing a string". Today Open Actions fails with arrays and objects cannot be interpolated as strings.
• runs-on group/labels mappings.
• String coercion of objects and arrays.
• Non-Linux runners for mixed-OS matrices.
• The other untracked gaps in the table above.

Assessment validation

• The 2026-09-30 and 2026-10-20 scans attributed failures with a YAML feature scanner. The 2026-11-03 rescan classifies the parser's own error messages instead. It left 20 files unresolved, for example:

  • A runs-on label such as 1ES.Pool=....
  • More than 25 workflow_call inputs.
  • needs on a missing job.

  Runtime gaps such as non-Linux runners, Docker actions, and caches were not assessed, so these counts are upper bounds on workflows that would run unchanged.

• In the 2026-11-03 rescan on e9134c6, every one of the 27 files blocked only by structured matrix values failed with ... values must be scalars. They come from 13 repositories: ComfyUI, llama.cpp, ollama, open-webui, codex, openinterpreter, prometheus, protobuf, RuView, rustdesk, tauri, tesseract, and whisper.cpp. Seven of them are workflow_call-only:

  • codex and openinterpreter rust-release-argument-comment-lint.yml.
  • prometheus fuzzing.yml.
  • protobuf test_cpp.yml, test_hpb.yml, and test_objectivec.yml.
  • rustdesk bridge.yml.

  The earlier scans also counted files from uv, bat, and storytold/photocraft.

• Contract probe on 0586283:

  • Official array-of-objects example: job "test" matrix axis "node" values must be scalars.
  • include entry with an object value: matrix include entry 1 values must be scalars.
  • fromJSON matrix with an axis of objects passed to EvaluateMatrix: job "build" matrix axis "target" values must be scalars.
  • include: [{OS: a, extra: yes}] with os: [a, b]: two combinations, both carrying OS: a and extra: yes.

• The probe on e9134c6 in Latest verdict reproduces each of these failures and also covers an axis of arrays and an object axis. The runner probe below covers nested runner expressions.

• The 2026-11-03 corpus contains 309 structured matrix values. Each encodes to at most 457 bytes of compact JSON; the largest is in protobuf test_upb.yml.

• None of the three corpora, including the 2026-11-03 corpus, has a file with include/exclude keys whose case differs from the axis name, or with axis names that differ only by case. The case-insensitivity contract is documented but has not been seen blocking a real workload.

• Runner probe on 87db1d2, repeated on 5c8d765 with a version-10 plan. A plan whose matrix is {job: {target, use-cross: true}, fuzz_test: [FuzzA, FuzzB]} was decoded as JSON into Plan.Matrix and evaluated in the runner's step and step-condition expression contexts ("matrix": plan.Matrix):

  • ${{ matrix.job.target }}, ${{ join(matrix.fuzz_test, ' ') }}, ${{ toJSON(matrix) }}, and the condition matrix.job.use-cross return the expected values.
  • Only bare ${{ matrix.job }} fails, with arrays and objects cannot be interpolated as strings, and string coercion is out of scope.
  • The controller already passes the typed combination to planning expressions, to plan.Matrix, and to reruns. The remaining work is in internal/workflow and in the fmt.Sprint-based string form described in the table above.

• The temporary probes ran through make test and were deleted. Repository files are unchanged.