Add Docker-based CI/CD for fast testing

.github/ISSUE_TEMPLATE/test_coverage_improvement.md

@@ -0,0 +1,58 @@
+---
+name: Test Coverage Improvement
+about: Track test coverage improvements for specific folders
+title: 'Improve test coverage for [FOLDER]'
+labels: 'testing, enhancement'
+assignees: ''
+---
+
+## Folder
+<!-- e.g., proteus.config, proteus.interior, etc. -->
+
+## Current Coverage
+<!-- Run: pytest --cov=src/proteus/[folder] --cov-report=term-missing -->
+```
+Current: X%
+Target: Y%
+```
+
+## Uncovered Lines
+<!-- From coverage report --show-missing -->
+```
+file.py: 10, 25-30, 45
+```
+
+## Test Strategy
+
+### Unit Tests Needed
+- [ ] Function: `function_name()` (lines X-Y)
+- [ ] Function: `another_function()` (lines X-Y)
+- [ ] Class: `ClassName` (lines X-Y)
+
+### Integration Tests Needed
+- [ ] Integration point: description
+- [ ] Workflow: description
+
+### Edge Cases
+- [ ] Error handling for X
+- [ ] Boundary conditions for Y
+- [ ] Invalid input handling
+
+## Implementation Plan
+
+1. [ ] Create test file: `tests/[folder]/test_[feature].py`
+2. [ ] Add fixtures in `conftest.py` (if needed)
+3. [ ] Write unit tests
+4. [ ] Write integration tests
+5. [ ] Run locally: `pytest tests/[folder]/`
+6. [ ] Verify coverage: `pytest --cov=src/proteus/[folder] --cov-report=html`
+7. [ ] Update documentation
+
+## Success Criteria
+- [ ] Coverage increases to target %
+- [ ] All new tests pass
+- [ ] No regressions in existing tests
+- [ ] CI pipeline passes
+
+## Notes
+<!-- Additional context, challenges, or considerations -->

.github/copilot-instructions.md

@@ -0,0 +1,74 @@
+# PROTEUS Ecosystem Copilot Guidelines
+
+You are an expert Scientific Software Engineer working on the PROTEUS ecosystem.
+
+## Ecosystem Structure
+
+PROTEUS is a coupled atmosphere-interior framework with a modular architecture:
+
+- **[PROTEUS](https://github.com/FormingWorlds/PROTEUS)** (main repository): Core coupling framework and orchestration
+- **[AGNI](https://github.com/nichollsh/AGNI)**: Radiative-convective atmospheric energy module (Julia)
+- **[SOCRATES](https://github.com/nichollsh/SOCRATES)**: Spectral radiative transfer code (Fortran)
+- **[CALLIOPE](https://github.com/FormingWorlds/CALLIOPE)**: Volatile in-/outgassing and thermodynamics module (Python)
+- **[JANUS](https://github.com/FormingWorlds/JANUS)**: 1D convective atmosphere module (Python)
+- **[MORS](https://github.com/FormingWorlds/MORS)**: Stellar evolution module (Python)
+- **[ARAGOG](https://github.com/FormingWorlds/aragog)**: Interior thermal evolution module based on T-P formalism (Python)
+- **[SPIDER](https://github.com/djbower/spider)**: Interior thermal evolution module based on T-S formalism (C)
+- **[VULCAN](https://github.com/FormingWorlds/VULCAN)**: Atmospheric chemistry module (Python)
+- **[ZEPHYRUS](https://github.com/FormingWorlds/ZEPHYRUS)**: Atmospheric escape module (Python)
+- **[Love.jl](https://github.com/FormingWorlds/Love.jl)**: Tidal evolution module (Julia)
+
+**Important:** Each module is maintained in its own GitHub repository but is typically cloned/installed within the PROTEUS directory structure for integrated development. When working on any module in the ecosystem, apply these guidelines consistently.
+
+## Scope of These Guidelines
+
+**These guidelines apply to ALL Python modules in the PROTEUS ecosystem.** Whether you are working in:
+- The main PROTEUS repository
+- A standalone module (CALLIOPE, JANUS, MORS, etc.)
+- Tests for any ecosystem component
+
+Follow the same standards for testing, coverage, code quality, and infrastructure.
+
+## Installation & Dependencies
+
+For installation instructions and dependency management across the ecosystem:
+- **Main installation guide:** `docs/installation.md` - Standard user and developer installation procedures
+- **Local machine setup:** `docs/local_machine_guide.md` - Platform-specific setup (macOS, Linux, Windows)
+- **Cluster setup:** `docs/kapteyn_cluster_guide.md` - HPC cluster configuration (see also `habrok_cluster_guide.md`, `snellius_cluster_guide.md`)
+
+When helping with installation or dependency issues, always reference these guides first. The `proteus install-all` command handles most submodule installations automatically. However, whenever possible, prefer the developer installation steps outlined in the installation guide for editable installs.
+
+## 1. Test Infrastructure & Organization
+- **Structure:** Tests MUST mirror the source code structure exactly. For every file in `src/<package>/`, create a corresponding `tests/<package>/test_<filename>.py`.
+- **Example:** `src/proteus/config/_config.py` → `tests/config/test_config.py`
+- **Discovery:** Use `pytest --collect-only` to verify test discovery before writing tests.
+- **Tools:** Run `bash tools/validate_test_structure.sh` to check if tests mirror source structure.
+- **Documentation:** See `docs/test_infrastructure.md` for full testing infrastructure details.
+
+## 2. Testing Standards (pytest)
+- **Framework:** Use `pytest` exclusively in the `tests/` directory.
+- **Coverage Tool:** Two equivalent approaches are supported:
+  - Local: `pytest --cov` (uses pytest-cov plugin, convenient)
+  - CI/Local: `coverage run -m pytest` (matches CI exactly, compatible with ratcheting)
+  - Choose based on preference; both work correctly.
+- **Speed:** Unit tests must run in <100ms. Aggressively mock heavy simulations, I/O, and external APIs using `unittest.mock`.
+- **Integration:** Mark slow tests (full simulation loops) with `@pytest.mark.slow`.
+- **Markers:** Use pytest markers: `@pytest.mark.unit` for unit tests, `@pytest.mark.integration` for integration tests.
+- **Floats:** NEVER use `==` for floats. Use `pytest.approx(val, rel=1e-5)` or `np.testing.assert_allclose`.
+- **Physics:** Ensure inputs are physically valid (e.g., T > 0K) unless testing error handling.
+
+## 3. Coverage Requirements
+- **Threshold:** Check `pyproject.toml` [tool.coverage.report] `fail_under` for current threshold.
+- **Automatic Ratcheting:** Coverage threshold automatically increases on main branch via `tools/update_coverage_threshold.py` (never decreases). See CALLIOPE for reference implementation.
+- **Reports:** Run `pytest --cov --cov-report=html` and inspect `htmlcov/index.html` for gaps.
+- **Analysis:** Use `bash tools/coverage_analysis.sh` to identify low-coverage modules needing tests.
+- **Quality Gate:** All PRs must pass the coverage threshold defined in CI (see `.github/workflows/proteus_test_quality_gate.yml`).
+
+## 4. Code Quality & Style
+- **Linting:** Follow `ruff` standards. Line length < 92 chars, max indentation 3 levels.
+- **Type Hints:** Use standard Python type hints.
+- **Docstrings:** Include brief docstrings describing the physical scenario.
+
+## 5. Safety & Determinism
+- **Randomness:** Explicitly set seeds (e.g., `np.random.seed(42)`) in tests.
+- **Files:** Do not generate tests that produce large output files (unless explicitly instructed); use `tempfile` or mocks.

.github/workflows/ci-nightly-science.yml

@@ -0,0 +1,158 @@
+name: CI - Nightly Science Validation
+
+# Purpose: Deep validation of scientific accuracy using pre-built Docker image
+# Runs comprehensive physics simulations to ensure correctness
+# Triggers: Nightly at 03:00 UTC (1 hour after Docker build completes)
+
+on:
+  schedule:
+    - cron: "0 3 * * *"  # Nightly at 03:00 UTC
+  workflow_dispatch:  # Allow manual trigger
+
+permissions:
+  contents: read
+  packages: read
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: formingworlds/proteus
+
+jobs:
+  science-validation:
+    name: Full Science Validation (@pytest.mark.slow)
+    runs-on: ubuntu-latest
+    timeout-minutes: 240  # 4 hours for comprehensive tests
+    container:
+      image: ghcr.io/formingworlds/proteus:latest
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+      options: --user root
+
+    steps:
+      - name: Checkout main branch
+        uses: actions/checkout@v4
+        with:
+          ref: main
+          fetch-depth: 0
+
+      - name: Overlay code onto container
+        run: |
+          echo "Copying code over container base..."
+          rsync -av --exclude='.git' --exclude='SPIDER' --exclude='socrates' --exclude='petsc' --exclude='AGNI' . /opt/proteus/
+          cd /opt/proteus
+          pip install -e ".[develop]" --no-deps
+
+      - name: Download test data if needed
+        run: |
+          cd /opt/proteus
+          # Uncomment if FWL_DATA needs to be populated
+          # proteus get stellar
+          # proteus get spectral --name Frostflow --bands 48
+
+      - name: Run slow integration tests
+        run: |
+          cd /opt/proteus
+          pytest -m slow -v --tb=long --maxfail=3 \
+            --cov=src \
+            --cov-report=term-missing \
+            --cov-report=xml \
+            --cov-report=html
+
+      - name: Upload coverage report
+        uses: codecov/codecov-action@v4
+        if: always()
+        with:
+          files: /opt/proteus/coverage.xml
+          flags: slow-integration-tests
+          name: nightly-science-coverage
+          fail_ci_if_error: false
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
+      - name: Upload HTML coverage
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: science-coverage-html
+          path: /opt/proteus/htmlcov/
+          retention-days: 30
+
+      - name: Upload simulation outputs
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: science-validation-outputs
+          path: |
+            /opt/proteus/output/
+            /opt/proteus/fwl_data/
+          retention-days: 30
+
+      - name: Notify on failure
+        if: failure()
+        run: |
+          echo "::error::Nightly science validation failed. Check logs and artifacts."
+          echo "This indicates potential scientific correctness issues."
+
+  integration-tests:
+    name: Integration Tests (Multi-module)
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    container:
+      image: ghcr.io/formingworlds/proteus:latest
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ secrets.GITHUB_TOKEN }}
+      options: --user root
+
+    steps:
+      - name: Checkout main branch
+        uses: actions/checkout@v4
+        with:
+          ref: main
+          fetch-depth: 0
+
+      - name: Overlay code onto container
+        run: |
+          echo "Copying code over container base..."
+          rsync -av --exclude='.git' --exclude='SPIDER' --exclude='socrates' --exclude='petsc' --exclude='AGNI' . /opt/proteus/
+          cd /opt/proteus
+          pip install -e ".[develop]" --no-deps
+
+      - name: Run integration tests
+        run: |
+          cd /opt/proteus
+          pytest -m integration -v --tb=long \
+            --cov=src \
+            --cov-report=term-missing \
+            --cov-report=xml \
+            --cov-report=html
+
+      - name: Upload coverage report
+        uses: codecov/codecov-action@v4
+        if: always()
+        with:
+          files: /opt/proteus/coverage.xml
+          flags: integration-tests
+          name: nightly-integration-coverage
+          fail_ci_if_error: false
+        env:
+          CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+
+      - name: Upload HTML coverage
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: integration-coverage-html
+          path: /opt/proteus/htmlcov/
+          retention-days: 30
+
+      - name: Upload integration outputs
+        uses: actions/upload-artifact@v4
+        if: failure()
+        with:
+          name: integration-test-failures
+          path: |
+            /opt/proteus/output/
+            /opt/proteus/tests/**/*.log
+          retention-days: 14