Skip to content

fix(workflow-runtime-core): bound step output and reduce oversized status reports - #1457

Open
alfredolopez80 wants to merge 5 commits into
devfrom
fix/workflow-status-output-bound
Open

alfredolopez80 wants to merge 5 commits into
devfrom
fix/workflow-status-output-bound

Conversation

@alfredolopez80

@alfredolopez80 alfredolopez80 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #1135, which is merged. git diff origin/dev shows only this PR's files.

Summary

Follow-up to the toolsCalled bounding in #1135, which left output unbounded.

  • The status reporter now cuts output to 65,536 characters, the largest preview WRC can be
    configured to store (MAX_WORKFLOW_STEP_OUTPUT_PREVIEW_MAX_CHARS, now defined from the shared
    STATUS_OUTPUT_MAX_CHARS), so the stored preview is the same prefix as before for every allowed
    configuration, unless the report needs the size reduction described below.
  • WRC extracts custom artifacts by parsing the complete output. When the reporter cuts the
    output, it extracts them first and sends them as outputArtifacts; WRC handles them like the
    artifacts it extracts from an uncut output: normalized, at most 21 (artifact plus 20 from
    artifacts[]), de-duplicated by name against status.artifacts. WRC reads them only when the
    report says the output was cut. Artifact extraction has one copy in
    @clerum/workflow-runtime-core; the reporter has its own serializer for the live output, and
    WRC keeps the one for the value it receives.
  • The reporter serializes a non-string output with JSON semantics, so an object output, including
    one that references the same object twice, is stored as before this PR. A BigInt or a toJSON
    that throws makes reportStepStatus throw, as before. Three kinds of output are stored
    differently from before this PR: an output that contains itself is stored with "[Circular]" at
    that point (before, the report threw); NaN and Infinity are stored as "null", and a
    function or a symbol as its String() text (before, no output was stored for these).
  • The reporter sends outputLength and outputTruncated so the stored preview metadata still
    describes the original output. WRC honours outputLength only when outputTruncated is true
    and the length is a safe integer at least as long as what it received, so a client can report a
    longer original output but never a shorter one, and cannot mark an output WRC cut as complete.
  • When the serialized step status body is over 256 KiB (half of WRC's 512 KiB limit), the
    reporter reduces it in a fixed order and stops as soon as it fits: cut error/failureReason
    to 8,192 characters; remove tool-call args and result previews from the last call
    backwards, keeping the artifact metadata WRC registers (also when mcp-host adds content to
    an artifact result); then halve output, not below 1,024 characters. Other fields are not
    reduced, so a report can still be over 256 KiB; it is sent anyway and logged with structural
    fields only (label, bodyBytes). After a halving WRC stores a preview shorter than its
    configured limit and records that length in outputPreviewMaxChars; a resumed run then
    interpolates the shorter preview.

Compatibility and deploy order

A coordinator without this change keeps working against the new WRC and stores the same step and
artifacts as before. A coordinator with this change is accepted by a WRC without it, but for an
output over 65,536 characters that WRC registers none of the artifacts declared in the output and
records outputLength 65,536. WRC and the default coordinator image are built from the same
sources and pinned to the same release in the WRC Deployment, so this needs a custom coordinator
image newer than WRC, or a WRC rollback while newer coordinators run. Roll out WRC before, or
together with, the coordinator image. No CRD schema change: the stored step fields already exist.

Behaviour change in inherited tests

Bounding the output is the point of this PR, so two tests from #1135 that asserted a full
150,000-character output now assert this PR's contract:

  • "bounds toolsCalled in step reports while keeping the full output" is now "bounds toolsCalled
    and output in step reports". It asserts the first 65,536 characters, outputLength: 150000 and
    outputTruncated: true, and measures the remaining body against the bounded output. Its
    toolsCalled assertions are unchanged.
  • "bounds multi-byte and control-character toolsCalled in the real request body" asserted the
    full 150,000-character output and a toolsCalled list over 200 KiB. With this PR the output is
    cut to 65,536 characters and the report is reduced to 256 KiB, so the test now asserts the cut
    output, that at least 40 of the 100 calls keep their previews, that every value sent is at most
    2,079 bytes, and that the body is between 200 and 256 KiB. fix(workflow): bound agent-step status reports and answer oversized WRC bodies with a readable 413 #1135's per-value byte bound is
    still what this test catches.

boundStepStatusBody is applied once per report; reportStepStatus is its only caller. Applied
twice it gives the same result when the output was not cut and no call past the first 50 had its
result replaced by a preview or removed.

Testing

All on head 5f2d9f324:

  • packages/workflow-runtime-core: npm run build clean; npm test 286/286 passed
    (16 files)
  • workflow-recipes: npm run build and npx tsc --noEmit clean; npm test 3416 passed,
    29 skipped (153 files passed, 4 skipped)
  • New tests: JSON-semantics serialization (cycles, shared references, toJSON, no-JSON
    values); artifact results kept through the reduction, including mcp-host's
    {success, artifact, content} shape; reduction from the last call backwards, only as much
    as needed; a body that still exceeds 256 KiB after fix(workflow): bound agent-step status reports and answer oversized WRC bodies with a readable 413 #1135's byte bounding, with a witness
    that the reduction ran; WRC storing the artifacts of a cut output like those of an uncut
    one; the recorded preview length
  • Each negative assertion was checked by a mutation that disables the behaviour; all 27
    mutations made the intended tests fail
  • scripts/minikube/t0.sh passed on this head
  • Minikube T1/T2: not run yet; packages/workflow-runtime-core is in t1Packages

Checklist

  • This is not a new third-party MCP server / WorkflowRecipe (those go to the registry)

🤖 Generated with Claude Code

alfredolopez80 and others added 2 commits November 12, 2026 20:29
…s report size

The status reporter now cuts step output to 65,536 characters, the largest
preview WRC can store, and keeps every step status body under 256 KiB by
degrading in a fixed order (error text, tool-call previews, then output).
Artifacts declared in a cut output are sent as outputArtifacts, and
outputLength/outputTruncated describe the original output; WRC clamps
both and merges the artifacts. Serialization and artifact extraction now
have one copy in workflow-runtime-core.

The inherited test that asserted the full output is sent now asserts the
bounded output, as an approved behaviour change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…limit' into fix/workflow-status-output-bound

* origin/fix/workflow-status-report-body-limit:
  test(workflow-recipes): assert the drain cap on what the server read
@zach88

zach88 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Took a deep pass through this one, scoped to this PR's own changes (e80a3afa2..e6dbd8f9f, i.e. on top of #1135). The design reads well: one copy of serialization and artifact extraction in runtime-core, a strict outputLength hint, and a byte ceiling that measures what is actually sent. The problems sit in the degradation path and at the reporter/WRC seam, where a few things now get stored differently than before, and some tests hand-write the reporter's body so they can't catch it.

Verdict: 🔴 Not ready to approve

  • Must resolve before approval: 1
  • Optional / non-blocking: 12

This is my read from the review, not a formal GH approval.

Must resolve before approval

1. [R1-H1] Degradation drops the standard artifact tool result, so the artifact vanishes from status.artifacts

packages/workflow-runtime-core/src/status-reporter/outputStatus.ts:141-155

When the body is still over 256 KiB after the output and error cuts, dropToolCallPreviews keeps a tool call's result only if isArtifactOnlyResult sees exactly the 2-key object { success: true, artifact }. But the result the internal artifact tools actually return is { success, artifact, content } (mcp-host/src/workflow/artifactOutput.ts:265-275, kept as is by mcp-host/src/workflow/internalToolProjection.ts:6-11). It's ~200 chars, so boundToolResultForStatus passes it through untouched (packages/workflow-runtime-core/src/status-reporter/toolCallStatus.ts:82), and then the degradation step drops it whole. Same for a JSON-string artifact result. WRC's parseUnknownArtifactResult would have extracted the artifact from all of these.

So the short, typical artifact result is the one that gets lost (a long one gets reduced to {success, artifact} earlier and survives). The UI and control-api read status.artifacts, so the run looks fine but the deliverable isn't listed, and nothing is logged. On #1135's head a body this size (~300 KB) went through whole and WRC registered the artifact, so this is new in the 256 to 512 KiB band.

It doesn't need anything exotic: an agentic step with an artifact tool call and around 40+ calls with CJK args/results, or a 64K-char Cyrillic or CJK output plus a lot of calls, crosses 256 KiB after the per-field caps. Plain ASCII/Spanish doesn't (218 to 238 KB in my runs).

Repro: feeding boundStepStatusBody a body built with the real boundToolsCalledForStatus (49 calls with CJK args/results, one of them an artifact result shaped like artifactResult's) gives a ~3 KB body with zero artifact results left. Same with a 64K Cyrillic output (283,656 B before degradation).

Suggested fix: in dropToolCallPreviews, decide with the same logic boundToolResultForStatus uses (parseArtifactToolResult + normalizeArtifactForStatus) and keep { success: true, artifact: <normalized> }, instead of requiring the exact 2-key shape.

The test at packages/workflow-runtime-core/tests/unit/output-status.test.ts:177 hand-writes { success: true, artifact }, which is exactly the one shape the code handles, so it can't catch this. It'd be worth deriving the results from artifactResult/projectInternalToolResult + boundToolsCalledForStatus and asserting what WRC would extract from the degraded body. The body's "keeping artifact metadata" is also not true for these shapes today.

Optional (non-blocking)

These can merge as follow-ups or at your call.

2. [R1-H2] (originally High, downgraded to Medium) "What WRC persists is unchanged for every allowed configuration" isn't true for 256 to 512 KiB bodies

packages/workflow-runtime-core/src/status-reporter/outputStatus.ts:110-137

WRC used to accept bodies up to 512 KiB whole. With the 256 KiB ceiling, bodies in between now lose tool-call args/result previews, and in some setups get failureReason cut to 8,192 chars and output halved below the configured preview. Reproduced by pushing the real producer's body through postStepStatus: 70k CJK chars + 50 calls with ~900-char args/results (303 KB) used to store args/result and now stores neither.

What it takes to trigger this:

  1. An agentic step, the only kind that sends toolsCalled (workflow-recipes/src/workflow/sdkRuntime.ts:300). Controlled by the recipe author; common.
  2. 25 to 50 tool calls in one step (MAX_STATUS_TOOL_CALLS = 50, packages/workflow-runtime-core/src/status-reporter/toolCallStatus.ts:7). Controlled by the LLM; occasional.
  3. Args and results over 1,024 chars and mostly CJK (3 bytes/char). With ASCII args it stays around 258 KB, right on the edge. Controlled by tenant data; rare outside CJK workflows.

Why it's unlikely to matter much: what's lost is 1 KiB diagnostic previews that were already truncated, and artifacts survive (apart from #1). The other two effects need conditions nobody runs. Halving the output below the preview needs WRC_WORKFLOW_STEP_OUTPUT_PREVIEW_MAX_CHARS above 32,768 (default 32 * 1024, workflow-recipes/src/config.ts:119) plus an output that's mostly control characters. Cutting failureReason needs an err.message over 256 KiB, and that text is still stored whole in workflowExecution.message. The 4th bullet of the body already describes the degradation; it's the "unchanged" sentence that contradicts it. The cheap fix is to reword that sentence. If you want the fidelity back, raise STATUS_REPORT_MAX_BYTES closer to 512 KiB with some margin.

3. [R1-M1] The 21st artifact of a cut output is lost (producer emits 21, WRC takes 20)

workflow-recipes/src/workflow/restEndpoints.ts:168 vs packages/workflow-runtime-core/src/status-reporter/outputStatus.ts:56-68

extractOutputArtifactsForStatus returns artifact plus artifacts.slice(0, 20), up to 21. WRC slices outputArtifacts to MAX_CUSTOM_ARTIFACTS_PER_STEP = 20, so for a cut output with artifact + 20 artifacts[] the last one is dropped. Extracting from the full output, the base stored all 21. Repro: reproduced end to end with the real producer and handler. Suggest one shared constant for the max, in one place. The test at restEndpoints.functional.test.ts:2266 certifies the 20 cap with a hand-written body (28 entries, including a string and /etc/passwd) the producer never emits.

4. [R1-M2] Dedupe by path is a new rule and only applies when the output was cut

workflow-recipes/src/workflow/restEndpoints.ts:167-174

Two artifacts with the same path and different names (inside artifacts[], or one from toolsCalled and one from the output) are both stored when the output is short and only one when it was cut. So the same step stores different things depending on its size. Repro: reproduced (base stored tool.md and r-0.md, now only tool.md). The name-based dedupe at :899 already absorbs the real duplicate (a cut output that still parses), so dropping the path Set looks enough.

5. [R1-M3] serializeOutputForStatus now marks shared references as [Circular], not just cycles

packages/workflow-runtime-core/src/status-reporter/outputStatus.ts:31-48

It now runs on the live object in the coordinator, and the WeakSet flags any repeated reference. Before, JSON.stringify of the body wrote shared subtrees in full, and WRC only ever saw a tree parsed from JSON. {a: s, b: s} is now stored as {"a":{...},"b":"[Circular]"}, and {meta:{first:a1}, artifacts:[a1]} goes from 1 artifact to 0, even with an uncut output. Repro: reproduced. This hits SDK/custom coordinators that pass in-memory objects (the agentic and snippet paths send JSON-derived outputs). Tracking ancestors on a stack instead of a seen-set would only catch real cycles.

6. [R1-M4] WRC tests hand-write the reporter body; the parity claim has only a trivial test

workflow-recipes/tests/unit/workflow/restEndpoints.functional.test.ts:2266, :2294

"adds valid outputArtifacts..." and "stores the original length of an output the coordinator cut" build outputArtifacts/outputLength/outputTruncated by hand. "A client without the new fields stores the same step and artifacts as before" only has one small case (:2241). A test that builds the body with boundStepStatusBody(raw) and compares against postStepStatus(raw) turned #2, #3 and #4 red, so it'd be worth keeping something like that (property-based with fast-check would be ideal: output size, artifact count and duplicates, tool-call sizes, preview at 1024/32768/65536).

7. [R1-L1] The 1,024-char floor isn't respected

outputStatus.ts:20-21, :123-134. The guard is checked before halving, so a 1,500-char output drops to 750 (reproduced). Math.max(MIN_DEGRADED_OUTPUT_CHARS, ...) or reword the comment/body.

8. [R1-L2] Cut-output artifacts are extracted from the raw object, not its serialized form

outputStatus.ts:82, :125. A Date createdAt gets restamped with the coordinator's clock (base kept the ISO string), and sizeBytes: NaN is now rejected (base got null and stored 0). Reproduced. extractOutputArtifactsForStatus(serialized) gives WRC's old semantics.

9. [R1-L3] A top-level Date output is now stored with quotes

outputStatus.ts:31-48. 2026-11-12T00:00:00.000Z becomes "2026-11-12T00:00:00.000Z" (reproduced), because the Date is now stringified on the client instead of arriving as a string.

10. [R1-L4] outputArtifacts is accepted even when outputTruncated !== true

restEndpoints.ts:166. A client can declare 21 in output plus 20 in outputArtifacts, 41 per step. No trust change (it could already declare them in output), but gating it on outputTruncated === true like outputLength would keep it tight.

11. [R1-L5] Rollout order: new coordinator against old WRC loses the artifacts of a cut output

With a cut output, an old WRC stores 0 of the declared artifacts and outputLength 65,536 (reproduced against a simulated old WRC). "A WRC without this change ignores them" is literally true, but it reads as if any order is safe. Worth adding "deploy WRC before the coordinator image" to the body.

12. [R1-L6] Idempotency and byte ceiling are only tested on a few hand-picked cases

packages/workflow-runtime-core/tests/unit/output-status.test.ts:145. Idempotency is checked on {output:'done', toolsCalled:[]}, which never reaches any degradation step. A few properties over generated bodies (fits when reachable, no mutation, idempotent when outputTruncated === false, every artifact WRC would extract from the original is still extractable) would cover what the body promises. fast-check isn't a devDependency of this package yet.

13. [R1-L7] Nit: the warning says "Step status report" for every label

packages/workflow-runtime-core/src/status-reporter/client.ts:35. report() emits it for the workflow status too (the test triggers it with label: 'workflow status'). "Status report exceeds the size ceiling" fits both.

What checked out clean

  • Security looks good. The status endpoint requires a clerum-wrc/control-api token with aud=clerum-wrc, the recipe/namespace binding and status_write; the only caller is that recipe's coordinator. outputArtifacts goes through the same normalizeArtifactForStatus (/output/ prefix, no .., no NUL, length caps) and doesn't let the caller register anything it couldn't already via output/toolsCalled. No prototype pollution (tested), and the new fields aren't persisted on the step.
  • Old coordinator against new WRC behaves exactly like the base: serializeOutputForStatus and extractOutputArtifactsForStatus are line-for-line the removed WRC functions.
  • Byte counting matches what report() sends (6-byte escapes included), the halving loop terminates, exactly 65,536 chars isn't cut, outputLength keeps the original length after halving, and the input isn't mutated. A real cycle in the output no longer blows up report().
  • The outputLength/outputTruncated validation holds on every edge I tried, and the CRD field is an integer with no maximum.
  • No CRD change is needed: outputTruncated, outputLength and outputPreviewMaxChars already exist in charts/clerum-crds/crds/workflowrecipe.yaml.
  • Images build the new exports: WRC, control-api and workflow-sdk use file:../packages/workflow-runtime-core, and the Dockerfiles compile it from src/.
  • The log line on an oversized body carries only label/bodyBytes.
  • Suites: runtime-core npm test 231/231 and tsc --noEmit clean; workflow-recipes touched tests 94/94 and tsc --noEmit clean. The only reds in the full workflow-recipes run were 3 in snippetRuntime.test.ts from a local environment guard, unrelated to this diff.

alfredolopez80 and others added 3 commits November 14, 2026 16:03
…tput-bound

Brings in #1135 (final head 9b7258f) and #1453. Resolved conflicts:
- workflow-recipes version: dev 0.1.505 + 1 = 0.1.506 (package.json and both lock lines).
- restEndpoints.functional.test.ts: kept both new describe blocks (A3's
  output-bound tests and #1135's byte-heavy tool-call tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts when reducing a report

Address the round-1 review of #1457 (zach88 R1-H1/H2/M4/L2-L6, blind
review A3-*) on top of #1135's byte-bounded tool-call previews:

- Serialize the step output with JSON semantics; only an object that
  contains itself becomes "[Circular]" (ancestor stack).
- When a report is reduced, keep the accepted artifact of every tool
  call (also when mcp-host adds `content`) and remove previews from the
  last calls first, only as much as needed.
- Extract artifacts of a cut output from the serialized text and cap
  them at MAX_REPORTED_OUTPUT_ARTIFACTS.
- Tests: an input that still exceeds 256 KiB after byte bounding, with a
  witness that the reduction ran (output-status), and #1135's reporter
  test asserted against the cut-output contract (status-reporter).

Orphans checked: isArtifactOnlyResult, dropToolCallPreviews; removed:
both (replaced by the artifact-preserving reduction, no remaining caller).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… of an uncut one

WRC side of the #1457 round-1 fixes (zach88 R1-H1, A3-M1 and the
preview findings):

- postStepStatus reads `outputArtifacts` only for an output the
  coordinator cut (`outputTruncated: true`), caps them at the shared
  MAX_REPORTED_OUTPUT_ARTIFACTS, and handles them like the artifacts it
  extracts from an uncut output.
- The stored `outputPreviewMaxChars` is the limit that produced the
  preview: WRC's, or the shorter length the coordinator cut to (OD-3).
- docs/crds/workflowrecipe.md: the preview-length sentence, and that
  tool-call previews are removed only while the body is still over
  256 KiB after error/failureReason are cut.

Orphans checked: MAX_CUSTOM_ARTIFACTS_PER_STEP; removed (replaced by the
core constant, no remaining reference).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alfredolopez80 alfredolopez80 changed the title fix(workflow-runtime-core): bound step output and guarantee the status report size fix(workflow-runtime-core): bound step output and reduce oversized status reports Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants