Skip to content

test(cli): set the fixture's env vars through monkeypatch so they cannot leak - #741

Merged
fas89 merged 1 commit into
mainfrom
fix/cli-test-env-leak
Oct 22, 2026
Merged

fas89 merged 1 commit into
mainfrom
fix/cli-test-env-leak

Conversation

@fas89

@fas89 fas89 commented Oct 22, 2026

Copy link
Copy Markdown
Collaborator

Description

The 3 tests/test_overlay_keyed_merge.py::TestValidateStrict failures on main are not
random. They are a test-isolation leak, and they appear whenever that file runs after
tests/cli/test_cli_output_keeps_brackets.py in the same worker.

How the leak worked:

  • tests/cli/conftest.py::test_environment wrote FLUID_ENV and FLUID_LOG_LEVEL
    straight into os.environ and restored a snapshot at teardown.
  • Autouse fixtures in the root conftest (_disable_copilot_self_eval) and in
    tests/cli/conftest.py (_reset_cli_process_state) create monkeypatch first. So
    monkeypatch.undo() ran after that restore.
  • A test that called monkeypatch.delenv or monkeypatch.setenv on either variable
    recorded the fixture's value as the original. The undo then put it back for the rest
    of the session.
  • test_cli_output_keeps_brackets.py's _clean_env delenvs every FLUID_* key.
    test_cli_smoke.py and test_binding_path_anchoring.py leaked the same way.
  • Every later subprocess test inherited FLUID_LOG_LEVEL=DEBUG. TestValidateStrict
    then saw the loader's overlay_patch_retargeted warning on stderr a second time
    (assert 2 == 1).

The DuckDB 2.0 port PR builds a clean subprocess env inside TestValidateStrict, which
fixes the symptom. This PR removes the cause: both variables now go through
monkeypatch.setenv, which records the true original. The snapshot restore stays, for
anything a test changes in os.environ directly. The comment on #740's _cli_env() no
longer blames xdist workers; the helper stays, since it also keeps out a debug switch a
developer exported.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would break existing functionality)
  • Documentation update
  • Refactor / code cleanup
  • New provider or provider enhancement

Documentation

  • Docs PR linked:
  • No docs needed — this change does not affect user-facing behaviour
    • Justification: test-only change (a pytest fixture and its regression test).
  • Docs TODO

Checklist

  • I have read the Contributing Guide
  • My code follows the project's coding standards
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass (pytest)
  • I have run ruff check and black with no errors
  • New maintained Python files include license headers (python scripts/add_license_headers.py; examples/** exempt)

Testing

  • New tests/cli/test_conftest_env_isolation.py. At module teardown it asserts that
    neither variable outlived the module. It fails on the old fixture
    ({'FLUID_ENV': 'test', 'FLUID_LOG_LEVEL': 'DEBUG'} outlived this module's tests),
    and it also fails when only one of the two variables is reverted. It passes here.
  • pytest tests/cli/test_cli_output_keeps_brackets.py tests/test_overlay_keyed_merge.py::TestValidateStrict -p no:randomly:
    3 failed on main, 20 passed here.
  • All of tests/cli: 2685 passed. The remaining failures happen the same way on
    main: sqlglot version skew in the local venv, plus subprocess and MCP-live timeouts
    under machine load.
  • Reviewed by an adversarial pass. It reproduced the mechanism with --setup-show. It
    found no new leak, either from keeping the snapshot restore beside monkeypatch or
    under xdist or random order. It asked for the second variable to be pinned too,
    which is done.

…not leak

tests/cli/conftest.py::test_environment wrote FLUID_ENV and
FLUID_LOG_LEVEL straight into os.environ and restored a snapshot at
teardown. Autouse fixtures in the root conftest and in tests/cli's own
create monkeypatch first, so monkeypatch's undo ran after that restore.
A test that called monkeypatch.delenv or monkeypatch.setenv on either
variable recorded the fixture's value as the original, and the undo put
it back for the rest of the session. test_cli_output_keeps_brackets.py's
_clean_env delenvs every FLUID_* key; test_cli_smoke.py and
test_binding_path_anchoring.py leaked the same way. Every later
subprocess test inherited FLUID_LOG_LEVEL=DEBUG, and
tests/test_overlay_keyed_merge.py::TestValidateStrict saw the loader log
its warning a second time: 3 failures whenever it ran after the brackets
file in one worker.

Both variables now go through monkeypatch, which records the true
original. The snapshot restore stays for anything a test changes in
os.environ directly. tests/cli/test_conftest_env_isolation.py checks, at
module teardown, that neither variable outlives the module; it fails on
the old fixture and on a revert of either variable alone.
@github-actions github-actions Bot added the tests Test coverage or test infrastructure changes label Oct 22, 2026
@fas89
fas89 merged commit acbcb5b into main Oct 22, 2026
33 checks passed
@fas89
fas89 deleted the fix/cli-test-env-leak branch October 22, 2026 12:02
@fas89 fas89 mentioned this pull request Oct 22, 2026
1 of 3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant