Skip to content

fix: pin encoding=utf-8 on text-mode open() reads in TupleFilesystemStoreBackend, InlineStoreBackend, FileDataContext - #12121

Open
CAOShurong wants to merge 1 commit into
fivetran:developfrom
CAOShurong:fix/issue-12120-utf8-pinned-reads
Open

fix: pin encoding=utf-8 on text-mode open() reads in TupleFilesystemStoreBackend, InlineStoreBackend, FileDataContext#12121
CAOShurong wants to merge 1 commit into
fivetran:developfrom
CAOShurong:fix/issue-12120-utf8-pinned-reads

Conversation

@CAOShurong

Copy link
Copy Markdown

Fixes #12120: great_expectations wrote store files as UTF-8 unconditionally but read them back with bare open() (locale-ambient). On hosts where the process locale encoding is not UTF-8 (e.g. Windows cp936), this raised UnicodeDecodeError on the very bytes great_expectations itself wrote.

Changes

  • tuple_store_backend.py:318 _get() open(filepath)open(filepath, encoding="utf-8")
  • inline_store_backend.py:222 _save_changes() write → open(..., encoding="utf-8")
  • file_data_context.py:168 _save_project_config() write → open(..., encoding="utf-8")
  • file_data_context.py:194 _load_file_backed_project_config() read → open(..., encoding="utf-8")

Verification

  • Reproduced the bug on a Windows host (codepage cp936): bare open() raised UnicodeDecodeError: 'gbk' codec can't decode byte 0x94 / 0xad on the non-ASCII UTF-8 bytes the write path emitted.
  • With encoding="utf-8" pinned on the four text-mode open() calls, round-trips succeed under forced PYTHONUTF8=0 / LC_ALL=C.

Regression tests

Added two tests that run under PYTHONUTF8=0 / LC_ALL=C to force the non-UTF-8 codepage:

  • test_tuple_store_backend_utf8_pinned_read_roundtrip in tests/data_context/store/test_store_backends.py
  • test_file_data_context_utf8_pinned_yml_load in tests/data_context/test_loading_and_saving_of_data_context_configs.py

Both fail against unpatched develop and pass with the fix.

AI-assisted: yes (Hermes General agent, autonomous loop round v6)

…toreBackend, InlineStoreBackend, FileDataContext

Fixes fivetran#12120: great_expectations wrote store files as UTF-8 unconditionally
but read them back with bare open() (locale-ambient). On hosts where the
process locale encoding is not UTF-8 (e.g. Windows cp936), this raised
UnicodeDecodeError on the very bytes great_expectations itself wrote.

Changes:
- tuple_store_backend.py:318  _get()  open(filepath) -> open(filepath, encoding="utf-8")
- inline_store_backend.py:222 _save_changes() write -> open(..., encoding="utf-8")
- file_data_context.py:168    _save_project_config() write -> open(..., encoding="utf-8")
- file_data_context.py:194    _load_file_backed_project_config() read -> open(..., encoding="utf-8")

Adds regression tests that force PYTHONUTF8=0 / LC_ALL=C in a subprocess
to trigger the non-UTF-8 codepage and assert the bug is fixed.

AI-assisted: yes (Hermes General agent, autonomous loop round v6)
@netlify

netlify Bot commented Aug 28, 2026

Copy link
Copy Markdown

👷 Deploy request for niobium-lead-7998 pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit 454d6dd

@github-actions github-actions Bot added the cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md label Aug 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Before we can merge this pull request, every committer needs to have signed our Contributor License Agreement (CLA).

We could not find a signed CLA for: @CAOShurong. Please sign the Individual Contributor License Agreement, or the Software Grant and Corporate Contributor License Agreement if you are contributing on behalf of your employer (see CLA.md for details).

Once resolved, comment @cla-bot check on this pull request to re-run the check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] TupleFilesystemStoreBackend reads store files with the ambient locale encoding while its write path pins UTF-8

1 participant