Plumb statistics object into ProgressThread for all engines

[wence-]

Description

We need this so that the communicators that we build have an active statistics object for keeping tack of communcation volume.

Previously we never noticed this because the only communication volume statistics we tracked were from the Shuffler that got its statistics from the buffer resource. However, it makes more sense to use the communicator since (for example) the allreduce implementation doesn't allocate and therefore doesn't have a buffer resource.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[wence-]

Needs rapidsai/rapidsmpf#1118

[pentschev]

Left another comment, and I think a cuDF-level test that performs a communicator-backed operation and observes its send/receive counter through gather_statistics() would be good to have, otherwise LGTM. Thanks Lawrence!

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 141bdca8-780f-48c6-8cb4-18776917196b

📥 Commits

Reviewing files that changed from the base of the PR and between 1ffbde5 and a098bf9.

📒 Files selected for processing (1)

• python/cudf_polars/tests/streaming/test_statistics.py

---

📝 Walkthrough

Summary by CodeRabbit

• New Features

  • Improved statistics and progress tracking consistency across execution engines, now aligned with the active runtime configuration.
  • Statistics objects are reused during engine lifecycle events instead of being recreated.

• Bug Fixes

  • Fixed cases where progress/statistics could be initialized or reset inconsistently across setup/reset paths.
  • Reset behavior now updates the enabled/disabled state without losing collected statistics state unexpectedly.

• Tests

  • Updated streaming statistics test expectations after clearing statistics.

• Documentation

  • Refreshed engine documentation to reflect the updated statistics behavior.

Walkthrough

Adds a reset_statistics_from_options helper in core.py and updates Dask, Ray, SPMD, and the default singleton engine to reuse existing Statistics objects while rebuilding or resetting statistics state.

Changes

Statistics reuse and reset

Layer / File(s)	Summary
reset_statistics_from_options helper python/cudf_polars/cudf_polars/engine/core.py	Adds a helper that enables or disables an existing Statistics instance based on Options, returning the mutated instance without clearing it.
Dask worker statistics wiring and reset python/cudf_polars/cudf_polars/engine/dask.py, python/cudf_polars/tests/streaming/test_statistics.py	Stores per-worker Statistics on _WorkerContext, threads that object through setup, reset, and statistics collection, and updates the streaming statistics test to accept the retained "event-loop-total" entry after clear=True.
Ray RankActor statistics reuse python/cudf_polars/cudf_polars/engine/ray.py	Wires self._rapidsmpf_statistics into ProgressThread during setup, adds a post-barrier _base_mr assertion, and replaces statistics recreation in reset with the reset helper followed by clear().
SPMDEngine statistics init and reset python/cudf_polars/cudf_polars/engine/spmd.py	Updates the class docstring and __init__ to construct or reset Statistics depending on whether a communicator is provided, and updates _reset to use the helper plus clear().
Default singleton engine ProgressThread python/cudf_polars/cudf_polars/engine/default_singleton_engine.py	Precomputes RapidsMPF options, derives Statistics from them, and passes a Statistics-backed ProgressThread and options into single_communicator.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly matches the main change: passing statistics objects into ProgressThread across engines.
Description check	✅ Passed	The description is directly about giving communicators an active statistics object to track communication volume.
Docstring Coverage	✅ Passed	Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[wence-]

/merge