Make combinatorial model memory safe

[bibymaths]

Motivation

• The combinatorial topology (MODEL=2) materialized dense hypercube transition arrays and an ns×n_sites bit matrix which caused exponential memory amplification and OOM failures for modest n_sites; the change implements conservative streaming/lazy strategies to avoid those allocations without changing model equations or objectives.
• Preserve exact transition enumeration order and numerical results for small combinatorial cases while preventing accidental large runs by adding explicit safety guards.
• Reduce retained memory in multistart and sensitivity workflows by streaming or bounding stored trajectories rather than accumulating full DataFrames in memory.

Description

• Implement lazy transition enumeration and compatibility gating in networkmodel/models.py by adding iter_random_transitions_for_sites, count_random_transitions_for_sites, and a guarded build_random_transitions that only materializes dense arrays for small proteins; the Numba combinatorial_rhs now enumerates phosphorylation edges on-the-fly in the same mask/site order.
• Add combinatorial memory guards, diagnostics, and a reusable per-step S_cache buffer in networkmodel/network.py; the constructor checks 2^n_sites per-protein and total state_dim and raises a clear MemoryError if unsafe, and the System now populates S_cache with the current site rates instead of preallocating a full site×time matrix.
• Replace dense bit-matrix extraction in networkmodel/simulate.py with streaming per-site aggregation via combinatorial_site_signals_streaming and free large temporaries immediately, preserving the returned DataFrame layout and values.
• Make the JAX/Diffrax RHS memory-safe in networkmodel/backend.py by removing dependence on global dense transition arrays and generating explicit phosphorylation transitions per protein from local n_states/n_sites metadata using JAX-compatible loops, and bound sensitivity retention in networkmodel/sensitivity.py so only the configured top curves are kept in memory during collection.
• Add unit tests tests/test_combinatorial_memory_safe.py validating tiny smoke case, streaming bit extraction against dense for n_sites=2..4, transition-order equivalence, dense-compatibility gating, and the memory guard; update docs docs/Combinatorial_Model_Memory_Issue.md describing implemented behavior and diagnostics.

Testing

• Compiled modified modules with python -m py_compile (checked networkmodel/models.py, networkmodel/network.py, networkmodel/simulate.py, networkmodel/backend.py, networkmodel/sensitivity.py, networkmodel/runner.py); compilation succeeded.
• Ran the new combinatorial test suite PYTHONPATH=. pytest -q tests/test_combinatorial_memory_safe.py and all tests passed (assertions check numerical equality to tight tolerances, e.g., 1e-12).
• Executed a focused integration subset including combinatorial/non-combinatorial backend checks (PYTHONPATH=. pytest -q tests/test_combinatorial_memory_safe.py tests/test_phoskintime_jax_multimodal.py::test_all_modes_detect_losses_objective_optimize_outputs_and_no_fake_missing_data) which passed.
• Ran the full test-suite (PYTHONPATH=. pytest -q) to surface unrelated environmental issues: 145 tests passed, 8 tests failed and 1 skipped; the failures are due to non-combinatorial/environmental factors (test expectations in the local config.toml for n_starts, missing optional plotting/notebook deps and an artifacts directory expectation) and are not caused by the combinatorial memory-safety changes.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1cb05b3ac5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep MODEL=2 S-rate exports two-dimensional

In the normal runner path I checked, runner.py calls export_S_rates after optimization for every model; for MODEL==2 that exporter still calls build_S_cache_into(sys.S_cache, ...) and treats sys.S_cache as a (total_sites, n_timebins) matrix before writing S_rates_picked.csv. Changing the cache here to a 1-D current-rate buffer makes combinatorial runs fail during result export rather than producing the expected output, so the exporter needs its own dense matrix or a streaming export path.

Useful? React with 👍 / 👎.