Skip to content

fix: include DataFrame content in pandas serde file name - #963

Open
digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/pandas-serde-content-hash
Open

digit50 wants to merge 1 commit into
apache:mainfrom
digit50:fix/pandas-serde-content-hash

Conversation

@digit50

@digit50 digit50 commented Oct 15, 2026

Copy link
Copy Markdown

closes #961

The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only. Two DataFrames with the same schema and shape, serialized with the same pandas_kwargs path, were written to the same file, so deserializing the earlier state returned the later DataFrame's data. Persisters and the tracking client serialize state at every step, so loading or forking from an earlier step could silently return the latest DataFrame.

Changes

  • serialize_pandas_df in burr/integrations/serde/pandas.py now also feeds the DataFrame content into the SHA-256 that names the file, using pd.util.hash_pandas_object(value, index=True).
  • For cells pandas cannot hash, such as lists, hash_pandas_object raises TypeError; in that case the parquet bytes of the DataFrame are hashed instead, so those DataFrames keep serializing as before.
  • The file name format (df_<64 hex chars>.parquet) is unchanged. Identical DataFrames still map to the same file.

How I tested this

  • Added test_serde_of_same_shape_dataframes_keeps_each_dataframe in tests/integrations/serde/test_pandas.py, parametrized over scalar cells and list cells. It serializes two same-shape DataFrames with different values to the same path, checks they get different file paths, and checks that State.deserialize of each returns its own DataFrame. Both cases fail on the current code (same file path) and pass with the change.
  • pytest tests/integrations/serde/test_pandas.py tests/integrations/serde/test_pickle.py tests/integrations/serde/test_pydantic.py: 14 passed.
  • pytest tests/core: 396 passed, 4 failed. The 4 failures are test_graphviz_display.py::test_visualize_dot_output, which needs the Graphviz dot binary that is not installed on my machine; they fail the same way without this change.

Notes

Files written by earlier versions keep their old names and are still read through the path stored in the serialized state, so existing persisted state loads as before.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

The pandas serializer named the parquet file after a hash of the columns, shape and dtypes only, so two same-shape DataFrames serialized to one path shared a file and the earlier state loaded the later data. The hash now also covers the DataFrame content, using pandas row hashes and falling back to the parquet bytes for cells pandas cannot hash, such as lists.

Closes apache#961
@github-actions github-actions Bot added the area/integrations External integrations (LLMs, frameworks) label Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/integrations External integrations (LLMs, frameworks)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pandas serde overwrites same-shape DataFrames, earlier state loads later data

1 participant