Skip to content

Lazy top-level namespace (PEP 562) for a cheaper import brightwind #631

Description

@dancasey-ie

Is your feature request related to a problem? Please describe.
brightwind/__init__.py eagerly runs from .load… import *, from .analyse.plot import *, etc., so import brightwind loads the entire dependency graph (matplotlib, gmaps, ipywidgets, IPython, colormap) even for a pure data transform that never plots. Any single optional dependency failing to import also breaks the whole package import. This inflates cold starts (e.g. AWS Lambda) and couples every consumer to notebook/GUI libraries.

Describe the solution you'd like
Load submodules lazily so import brightwind is cheap and heavy/optional deps load only when their namespace is actually touched. The flat public API stays unchanged. This is the Scientific Python "SPEC 1 — Lazy Loading of Submodules and Functions" convention (https://scientific-python.org/specs/spec-0001/), already used by scikit-image, SciPy, napari, etc.

Two implementation routes:

Option A — use lazy_loader (the scientific-python reference implementation).
https://github.com/scientific-python/lazy-loader collapses the whole __init__ to:

# brightwind/__init__.py
import lazy_loader as lazy
__getattr__, __dir__, __all__ = lazy.attach_stub(__name__, __file__)

with the public names declared in a brightwind/__init__.pyi stub — the stub also gives editors/mypy their static view, so no hand-written TYPE_CHECKING block is needed.

Caveat — verify before adopting: lazy_loader is a small project (~208 GitHub stars at the time of writing). Before taking it on as a runtime dependency, confirm it is actively maintained, still zero-dependency (so it adds no weight to the Lambda image), licence-compatible, and that attach_stub behaves correctly under the Lambda container / pip install --target layout.

Option B — hand-rolled __getattr__ (no new dependency).
If lazy_loader doesn't pass verification, the same behaviour can be implemented directly with PEP 562 (Python >= 3.7):

# brightwind/__init__.py (lazy, no dependency)
import importlib
from typing import TYPE_CHECKING

__version__ = "2.7.0"

# public name -> submodule providing it (generate from each submodule's __all__)
_LAZY = {
    "MeasurementStation":                ".load.station",
    "Correl":                            ".analyse.correlation",  # module alias
    "Shear":                             ".analyse.shear",        # ipywidgets/IPython
    "plot_meas_station_on_gmap":         ".utils.gis",            # gmaps
    "distance_between_points_haversine": ".utils.gis",            # pure maths
    # ... remaining load / analyse / transform / export names ...
}

def __getattr__(name):
    modpath = _LAZY.get(name)
    if modpath is None:
        raise AttributeError(f"module 'brightwind' has no attribute {name!r}")
    module = importlib.import_module(modpath, __name__)
    attr = module if name == "Correl" else getattr(module, name)
    globals()[name] = attr   # cache: __getattr__ not called again for this name
    return attr

def __dir__():
    return sorted(set(globals()) | set(_LAZY))

if TYPE_CHECKING:            # never executed at runtime; read only by static analysers
    from .load.station import MeasurementStation
    from .analyse.shear import Shear
    from .utils.gis import plot_meas_station_on_gmap, distance_between_points_haversine
    # ... mirror the rest of _LAZY ...

Result (either option) — the common data-only path stops importing the heavy stack:

import sys, brightwind as bw
"matplotlib" in sys.modules              # False after import brightwind
ms = bw.MeasurementStation(model)        # imports only .load.station (+ its deps)
"gmaps" in sys.modules                   # False
bw.plot_meas_station_on_gmap(ms)         # only now is gmaps imported

Editor / IDE / Jupyter support
Lazily-bound names can't be followed by static analysis unless declared, so the static view must be provided — the .pyi stub (Option A) or the TYPE_CHECKING block (Option B) — alongside __dir__ for the dynamic path:

  • __dir__ -> the dynamic/runtime path: dir(bw), the plain python REPL, and IPython completion when Jedi is disabled.
  • .pyi stub / TYPE_CHECKING block -> the static path: VS Code (Pylance/Pyright), mypy in CI, and JupyterLab's Jedi completer (Jedi treats TYPE_CHECKING as true). Both cost nothing at runtime.

Jupyter specifics:

  • The kernel never executes the static-only declarations (TYPE_CHECKING is False at runtime; .pyi files are not imported) — they only affect tooling.
  • Autocomplete is frontend-dependent: VS Code notebooks (Pylance) and JupyterLab/Jedi generally resolve the declared names; with Jedi disabled (%config IPCompleter.use_jedi = False) completion falls back to __dir__.
  • Shift-Tab signatures, bw.Shear? and help(...) are runtime introspection — they trigger the real __getattr__, so they return correct docstrings/signatures regardless.

Describe alternatives you've considered

  • Leave as-is — acceptable once optional deps are gated (see the notebook/GUI-deps issue), but eager import still slows cold starts and lets a single failing optional dep break import brightwind.
  • Submodule-only laziness (drop the flat re-exports, require bw.analyse.plot.…) — smaller __init__, but breaks the existing public API.

Additional context
Lower priority: most of the practical benefit comes from making optional dependencies lazy/optional (the notebook/GUI-deps and colormap issues). This change is mainly an import-time / cold-start improvement plus a robustness win. Whichever route is chosen, the public-name list (.pyi stub or _LAZY + TYPE_CHECKING mirror) should be generated from each submodule's __all__ so it cannot drift.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions