Preserve dataframe write step when a recipe wrangle strips real writes - #1158
Merged
Conversation
mborodii-prog
requested review from
ebhills and
thomasstvr
and removed request for
thomasstvr
September 1, 2026 07:42
ebhills
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix
write:being ignored inside a recipe used as a wrangleProblem
When a recipe is nested as a wrangle step (
wrangles: - recipe: ...), itswrite:section was silently discarded before the nested recipe ran - includingwrite: - dataframe: columns: [...], which has no external side effects at all.Its only job is to define what the nested
recipe.run()call returns. With itstripped, the nested recipe fell back to returning every column instead of the
requested subset.
Root cause
wrangles/recipe_wrangles/main.py::recipe()calledrecipe_object.pop('write', None)unconditionally before running the nestedrecipe. That was added in #1096 to stop a real write (
file,memory,s3,etc.) from firing once per chunk when the recipe wrangle is nested inside
batch:- which would otherwise produce duplicate/partial output each time thewrangle runs on a chunk. The fix was correct for real writes, but too broad: it
also dropped the
dataframewrite type, which never touches anything external.Fix
Only strip write entries that aren't
dataframe. Real, side-effecting writesare still removed (so the
#1096guarantee holds), butwrite: - dataframe:is preserved so it can shape the nested recipe's return value as intended.
Real writes nested inside a batched recipe wrangle are still correctly
stripped -
partial_recipe_writenever appears in this example, only theouter recipe's own write does:
Not a bug, for contrast:
recipe:used as a write connectorA separate report described
write: - recipe: ..."ignoring the recipealtogether" - that's a different code path
(
wrangles/connectors/recipe.py::write()), not touched by this fix, and it'salready working as intended. Every write connector in this codebase is
terminal: writing to
file,s3,memory, etc. never feeds back into theouter recipe's returned dataframe, and
recipe:as a write is no exception.The nested recipe's
create.columngenuinely runs (confirmed via logs), butwrite: - dataframe:only sets a return value, and the outerwrite()connector - like every write connector - never reads or exposes that return
value. Give the nested recipe's own
write:a real target instead(
memory,file, ...) to get a usable result: