Repository navigation
Conversation
The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only, so two same-shape DataFrames serialized to one path shared a file and the earlier state loaded the later data. The hash now also covers the DataFrame content, using pandas row hashes and falling back to the parquet bytes for cells pandas cannot hash, such as lists. Closes apache#961
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #961
The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only. Two DataFrames with the same schema and shape, serialized with the same
pandas_kwargspath, were written to the same file, so deserializing the earlier state returned the later DataFrame's data. Persisters and the tracking client serialize state at every step, so loading or forking from an earlier step could silently return the latest DataFrame.Changes
serialize_pandas_dfinburr/integrations/serde/pandas.pynow also feeds the DataFrame content into the SHA-256 that names the file, usingpd.util.hash_pandas_object(value, index=True).hash_pandas_objectraisesTypeError; in that case the parquet bytes of the DataFrame are hashed instead, so those DataFrames keep serializing as before.df_<64 hex chars>.parquet) is unchanged. Identical DataFrames still map to the same file.How I tested this
test_serde_of_same_shape_dataframes_keeps_each_dataframeintests/integrations/serde/test_pandas.py, parametrized over scalar cells and list cells. It serializes two same-shape DataFrames with different values to the same path, checks they get different file paths, and checks thatState.deserializeof each returns its own DataFrame. Both cases fail on the current code (same file path) and pass with the change.pytest tests/integrations/serde/test_pandas.py tests/integrations/serde/test_pickle.py tests/integrations/serde/test_pydantic.py: 14 passed.pytest tests/core: 396 passed, 4 failed. The 4 failures aretest_graphviz_display.py::test_visualize_dot_output, which needs the Graphvizdotbinary that is not installed on my machine; they fail the same way without this change.Notes
Files written by earlier versions keep their old names and are still read through the path stored in the serialized state, so existing persisted state loads as before.
Checklist