[OpenSTEF 4.0] Fix API reference completeness and structure

[egordm]

As a developer integrating or extending OpenSTEF
I want API docs that cover the full public surface with navigable hierarchy
So that I can discover and understand the library without reading source code

---

❗Priority

The API reference is missing approximately 40% of the public API surface. Specifically:

• openstef_meta (entire package — models, presets, utils) has zero pages
• openstef_models.transforms renders as an empty page — 24 public classes (OutlierHandler, Imputer, Scaler, etc.) are invisible
• 5 real openstef_models submodules (workflows, presets, integrations, utils, mixins) are undocumented
• Pydantic Field(description=...) values (painstakingly written) are not rendered anywhere

---

Root cause analysis

Problem	Root cause	Evidence
transforms page empty	__all__ = ['energy_domain', 'general', ...] exports module references, not classes. Autosummary can't classify ModuleType objects as functions/classes/attributes, so template variables are all empty	Generated stub contains only automodule directive — no classes/functions sections
Phantom pipelines, exceptions	Listed in api/index.rst under openstef_models but modules don't exist (pipelines → renamed to workflows)	WARNING: [autosummary] failed to import openstef_models.exceptions (×2 build warnings)
openstef_meta absent	Not listed in api/index.rst. Package IS importable in the docs venv — just never added to the RST	uv pip show openstef-meta → installed; api/index.rst has no entry
Pydantic fields invisible	No autodoc-pydantic extension. Standard autodoc shows field name + type but discards Field(description=...)	BacktestConfig source has description="Time interval between..." → HTML shows only prediction_sample_interval: timedelta
Sidebar noise	collapse_navigation: False with 95 generated pages + navigation_depth: 3	All items expanded simultaneously in left nav
Template hack	package_overview.rst hardcodes if fullname == 'openstef_core.datasets' to inject mixins submodule	Won't scale; transforms has the exact same problem

---

Definition of Done

✅ Acceptance criteria

Coverage:

• openstef_meta section added to API index (models, presets, utils)
• openstef_models.transforms lists all 5 submodules and their 24 public classes
• Missing modules added: workflows (2 classes), presets (1 class + 1 function), integrations (2 submodules), utils (4 submodules), mixins (3 classes)
• Phantom entries removed: pipelines, exceptions
• Zero autosummary-related build warnings

Pydantic rendering:

• Field(description=...) values shown alongside field definitions in HTML
• model_config ClassVar hidden from field listings (noise)
• Implementation: ~30 LOC autodoc-process-docstring hook preferred over autodoc-pydantic dependency (semi-maintained, single maintainer, last commit 2023)

Navigation:

• collapse_navigation: True (sections collapsed by default)
• Package → module → class hierarchy visible (not flat list)

Template:

• package_overview.rst handles submodule-exporting packages generically (no hardcoded module names)
• Recursive detection: if __all__ members are ModuleType, treat as submodules

📄 Documentation criteria

• The API reference itself IS the deliverable
• Module/package docstrings rendered as overview text

🧪 Test criteria

• poe docs builds with 0 autosummary/autodoc warnings
• Spot-check: openstef_models.transforms.general page shows OutlierHandler, Imputer, Scaler with class docstrings
• Spot-check: BacktestConfig page shows "Time interval between prediction samples..."
• Spot-check: openstef_meta.presets page renders

⌛ Dependencies

• docs: integrate myst-nb notebook rendering infrastructure (#884) #885 (myst-nb infra) — merged or rebased onto

🚀 Releasing

Part of v4.0.0 documentation

---

Technical approach (suggested)

1. Fix api/index.rst: Add openstef_meta section; replace pipelines→workflows, remove exceptions; add missing modules
2. Fix template: In package_overview.rst, detect module-type __all__ members generically (or explicitly list transforms submodules in api/index.rst as individual entries)
3. Pydantic hook (in conf.py): autodoc-process-docstring event that reads model_fields and appends field descriptions to the docstring
4. Nav config: Set collapse_navigation: True, reduce navigation_depth to 2