Skip to content

Preserve dataframe write step when a recipe wrangle strips real writes - #1158

Merged
ebhills merged 1 commit into
mainfrom
fix/Sub-recipe-outputs-are-being-ignored
Sep 1, 2026
Merged

Preserve dataframe write step when a recipe wrangle strips real writes#1158
ebhills merged 1 commit into
mainfrom
fix/Sub-recipe-outputs-are-being-ignored

Conversation

@mborodii-prog

Copy link
Copy Markdown
Contributor

Fix write: being ignored inside a recipe used as a wrangle

Problem

wrangles:
  - recipe:
      wrangles:
        - create.column:
            output: Col3
            value: Val3
      write:
        - dataframe:
            columns:
              - Col2
              - Col3

When a recipe is nested as a wrangle step (wrangles: - recipe: ...), its
write: section was silently discarded before the nested recipe ran - including
write: - dataframe: columns: [...], which has no external side effects at all.
Its only job is to define what the nested recipe.run() call returns. With it
stripped, the nested recipe fell back to returning every column instead of the
requested subset.

>>> wrangles.recipe.run("""
... read:
...   - test:
...       rows: 1
...       values:
...         Col1: Val1
...         Col2: Val2
... wrangles:
...   - recipe:
...       wrangles:
...         - create.column:
...             output: Col3
...             value: Val3
...       write:
...         - dataframe:
...             columns: [Col2, Col3]
... """)
   Col1  Col2  Col3          # before fix: write: - dataframe: was ignored
0  Val1  Val2  Val3

Root cause

wrangles/recipe_wrangles/main.py::recipe() called
recipe_object.pop('write', None) unconditionally before running the nested
recipe. That was added in #1096 to stop a real write (file, memory, s3,
etc.) from firing once per chunk when the recipe wrangle is nested inside
batch: - which would otherwise produce duplicate/partial output each time the
wrangle runs on a chunk. The fix was correct for real writes, but too broad: it
also dropped the dataframe write type, which never touches anything external.

Fix

Only strip write entries that aren't dataframe. Real, side-effecting writes
are still removed (so the #1096 guarantee holds), but write: - dataframe:
is preserved so it can shape the nested recipe's return value as intended.

>>> wrangles.recipe.run("""
... read:
...   - test:
...       rows: 1
...       values:
...         Col1: Val1
...         Col2: Val2
... wrangles:
...   - recipe:
...       wrangles:
...         - create.column:
...             output: Col3
...             value: Val3
...       write:
...         - dataframe:
...             columns: [Col2, Col3]
... """)
   Col2  Col3                # now: only the requested columns are returned
0  Val2  Val3

Real writes nested inside a batched recipe wrangle are still correctly
stripped - partial_recipe_write never appears in this example, only the
outer recipe's own write does:

wrangles:
  - batch:
      batch_size: 100
      wrangles:
        - recipe:
            wrangles:
              - convert.case:
                  input: header1
                  case: upper
            write:
              - memory:
                  id: partial_recipe_write   # still stripped, never runs
write:
  - memory:
      id: final_recipe_write                # this is the only write that fires

Not a bug, for contrast: recipe: used as a write connector

A separate report described write: - recipe: ... "ignoring the recipe
altogether" - that's a different code path
(wrangles/connectors/recipe.py::write()), not touched by this fix, and it's
already working as intended. Every write connector in this codebase is
terminal: writing to file, s3, memory, etc. never feeds back into the
outer recipe's returned dataframe, and recipe: as a write is no exception.

write:
  - recipe:
      wrangles:
        - create.column:
            output: Col3
            value: Val3
      write:
        - dataframe:            # has no external target - nothing observable happens
            columns: [Col2, Col3]

The nested recipe's create.column genuinely runs (confirmed via logs), but
write: - dataframe: only sets a return value, and the outer write()
connector - like every write connector - never reads or exposes that return
value. Give the nested recipe's own write: a real target instead
(memory, file, ...) to get a usable result:

write:
  - recipe:
      wrangles:
        - create.column:
            output: Col3
            value: Val3
      write:
        - memory:
            id: recipe_write_result   # retrievable via wrangles.connectors.memory.dataframes

@mborodii-prog
mborodii-prog requested review from ebhills and thomasstvr and removed request for thomasstvr September 1, 2026 07:42
@mborodii-prog mborodii-prog linked an issue Sep 1, 2026 that may be closed by this pull request
@ebhills
ebhills merged commit ae407aa into main Sep 1, 2026
13 checks passed
@ebhills
ebhills deleted the fix/Sub-recipe-outputs-are-being-ignored branch September 1, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sub recipe outputs are being ignored

2 participants