Skip to content

[MAINTENANCE] Make scipy an optional extra and drop redundant cryptography pin - #12083

Draft
dataders wants to merge 1 commit into
developfrom
dataders/trim-optional-scipy-cryptography-deps
Draft

[MAINTENANCE] Make scipy an optional extra and drop redundant cryptography pin#12083
dataders wants to merge 1 commit into
developfrom
dataders/trim-optional-scipy-cryptography-deps

Conversation

@dataders

Copy link
Copy Markdown
Collaborator

Summary

  • Moves scipy (82MB) out of the default install and into a new great_expectations[scipy] extra. It's only used by three files for scipy.stats (KL-divergence and KS-test expectations). A lazy-import wrapper in great_expectations/compatibility/scipy.py (same pattern as the existing compatibility/pyspark.py/compatibility/snowflake.py) gives a clear error pointing at the scipy extra if those specific expectations are used without it installed.
  • Drops cryptography (13MB) from requirements.txt entirely. It was already function-local imported in sqlalchemy_execution_engine.py for Snowflake key-pair auth (no eager cost), and snowflake-connector-python already brings it in transitively for anyone on the snowflake extra — the top-level pin was pure redundant weight.
  • Net effect: ~95MB smaller default install, no functional change for existing users of either dependency.

Test plan

  • great_expectations and all three scipy-dependent modules (expect_column_kl_divergence_to_be_less_than.py, column_bootstrapped_ks_test_p_value.py, column_parameterized_distribution_ks_test_p_value.py) import cleanly with scipy simulated as absent (verified via sys.modules/builtins.__import__ patching)
  • Existing KL-divergence integration test suite passes with scipy installed (tests/integration/data_sources_and_expectations/expectations/test_expect_column_kl_divergence_to_be_less_than.py, 9/9)
  • Full tests/expectations, tests/execution_engine, tests/datasource/fluent unit sweep: 2887 passed, 0 failed
  • great_expectations[scipy] extra confirmed generated correctly via setup.py::get_extras_require()

🤖 Generated with Claude Code

…ptography pin

scipy is 82MB and only used by three files for scipy.stats (KL-divergence and
KS-test expectations); it's now installed via `great_expectations[scipy]`
instead of every install, with a NotImported-style lazy wrapper in
compatibility/scipy.py giving a clear error if those specific expectations are
used without it. cryptography is dropped outright: it was already
function-local imported for Snowflake key-pair auth, and snowflake-connector-python
already brings it in transitively for anyone on the snowflake extra.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@netlify

netlify Bot commented Aug 20, 2026

Copy link
Copy Markdown

Deploy Preview for niobium-lead-7998 ready!

Name Link
🔨 Latest commit e871447
🔍 Latest deploy log https://app.netlify.com/projects/niobium-lead-7998/deploys/6a876ca11ae7b60008e095bc
😎 Deploy Preview https://deploy-preview-12083.docs.greatexpectations.io
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the pull request, @dataders! 👋

This change touches files that usually mean new data source or execution engine support:

  • great_expectations/compatibility/scipy.py — a new compatibility module (a new optional third-party dependency)
  • reqs/requirements-dev-scipy.txt — a new backend requirements file

Changes like that need an RFC agreed before implementation, so the design discussion happens before you invest in code. Sorry if this arrives after the fact — the check is here so the next contributor finds out at the right moment.

To resolve this check, add one of these lines to the pull-request description:

  • RFC: <link to the accepted discussion> — if an RFC exists or you open one now
  • No RFC needed: <reason> — if this isn't actually new backend support (for example, a bug fix that happens to touch these paths)

Either answer satisfies the check. A maintainer will pick it up from there.

@github-actions github-actions Bot added the cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Before we can merge this pull request, every committer needs to have signed our Contributor License Agreement (CLA).

We could not find a signed CLA for: @dataders. Please sign the Individual Contributor License Agreement, or the Software Grant and Corporate Contributor License Agreement if you are contributing on behalf of your employer (see CLA.md for details).

Once resolved, comment @cla-bot check on this pull request to re-run the check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-not-signed https://github.com/fivetran/great_expectations/blob/develop/CLA.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant