Scientific evaluation process

changelog.md

@@ -63,6 +63,7 @@ v1.0.0
 - [2925](https://github.com/RuminantFarmSystems/RuFaS/pull/2925) - [minor change] [NoInputChange] [NoOutputChange] Fix the `graph_and_report` option in report_generation.py.
 - [2907](https://github.com/RuminantFarmSystems/RuFaS/pull/2907) - [minor change] [NoInputChange] [OutputChange] Fix the FarmGrownFeed Emissions unit issue. The mirror issue of [Fix FarmGrownFeed Emissions Unit on test #2908](https://github.com/RuminantFarmSystems/MASM/pull/2908) to update `dev`.
 - [2934](https://github.com/RuminantFarmSystems/RuFaS/pull/2934) - [minor change] [NoInputChange] [NoOutputChange] [Animal][Reproduction] Refactor `Reproduction.execute_cow_ed_protocol()`.
+- [2840](https://github.com/RuminantFarmSystems/RuFaS/pull/2840) - [minor change] [Docs] [NoInputChange] [NoOutputChange] Adds page documenting workflow to acheive version update.
 
 
 ### v1.0.0

docs/_src/_wiki/Scientific-Review-Process.rst

@@ -0,0 +1,96 @@
+Model Review Process
+====================
+
+This document defines the process for reviewing the RuFaS model prior to a new version release. It complements the branching strategy, versioning policy, and maintainer workflow documentation.
+
+There are two main phases of the model review process:
+
+- Preparation and Methodology Review
+- Evaluation and Reporting
+
+Preparation and Methodology Review
+---------------------
+
+These steps may begin as soon as a new version release has been initiated and before merging development changes into the ``test`` branch.
+
+
+Update Evaluation Farm Dataset
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Ensure the evaluation farm input files are up to date and run without error on ``dev`` or on ``test`` after merging ``dev`` to ``test``.
+
+Update Desired Outputs
+^^^^^^^^^^^^^^^^^^^^^^
+
+Update all outputs required for evaluation, including:
+
+- RuFaS output filters
+- Post-processing scripts
+- Derived or summary outputs
+
+Update Benchmarks
+^^^^^^^^^^^^^^^^^
+
+- Review and update benchmark datasets and expectations as needed.
+- Ensure benchmarks reflect current model scope and scientific understanding.
+
+Evaluation and Reporting
+---------------------
+
+These steps occur after merging into the ``test`` branch.
+
+Run Model Simulations
+^^^^^^^^^^^^^^^^^^^^^
+
+- Run evaluation farms using the updated model.
+- Compile summarized results including comparison with previous results and benchmarks. 
+- Responsibility: Scientific Director
+
+
+Review Results Summary
+^^^^^^^^^^^^^^^^^^^^^^
+
+- Share results with Maintainer SMEs.
+- At least one SME from each module must review before proceeding.
+
+Iterate
+^^^^^^^
+
+- Repeat simulation and evaluation steps until results are acceptable.
+
+
+Reporting and Review
+--------------------
+
+Prepare Report
+^^^^^^^^^^^^^^
+
+- Summarize:
+
+  - Key results
+  - Comparisons to benchmarks and prior versions
+  - Notable changes
+
+Review Report
+^^^^^^^^^^^^^
+
+- Similar to above, at least one SME from each module should review the report.
+- Additional review from external or Contributor SMEs is recommended.
+
+Approval Criteria
+-----------------
+
+Proceed toward release when:
+
+- Each module is represented in both results review and report review.
+- Model performance is acceptable relative to:
+
+  - Benchmarks
+  - Prior versions
+  - Scientific expectations
+
+Notes
+-----
+
+- This process is iterative and may be adapted as needed.
+- Automation of datasets, outputs, and benchmarking workflows is encouraged to improve reproducibility and efficiency.

docs/_src/_wiki/Version-Scientific-Evaluation.rst

@@ -0,0 +1,92 @@
+Maintainer Team Workflow and Communication during a Version Update
+===================================================================
+
+This page describes the process that RuFaS Maintainers will use to communicate
+and execute a version update.
+
+In addition to the updated model and scientific documentation in the repository
+codebase, additional deliverables of this process for **Major** and **Minor**
+updates are:
+
+- A version release note that includes:
+
+  - A high-level summary of the outcomes of the SME Review
+  - A summary of the changelog
+
+- A detailed technical report of the SME Review
+
+
+
+Before Merging ``dev`` to ``test``
+------------------------------
+
+**Major** and **Minor** updates:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+1. RuFaS Maintainers meet to discuss what work **will and will not** be
+   included in the release and the proposed timeline.
+
+2. The PML reviews and approves pending work to be included and excluded
+   from the version update, with additional Maintainer input as needed.
+
+3. The PML and team leaders create a GitProject for the version release and
+   add pending issues/PRs that must be completed before the version update
+   (before ``dev → test``). If a Maintainer identifies additional work that would benefit
+   the version release after the project is created, issues/PRs should be added to
+   the version release project or discussed with the PML at the members discretion. 
+
+4. SME Maintainers review the scientific evaluation process and criteria from the
+   previous version release and update as necessary.
+
+
+**Patch** Updates
+^^^^^^^^^^^^^^^^^^
+
+Patch updates do not require a full project. Bug fixes required in the patch should be collected 
+as sub-issues under a **Patch version update parent issue**.
+
+
+After ``test`` is updated
+-------------------------
+
+RuFaS SME Maintainers evaluate integrated model performance:
+1. The Scientific Director runs Evaluation Farm Scenarios on ``test`` and compiles their outputs.
+2. SME Maintainers review model outputs Evaluation Farms on ``test``.
+2. Outputs are collected and summarized for the SME Review process
+   (link to upcoming SME Review page).
+3. SME Maintainers conduct the SME Review led by the Scientific Director.
+
+During this stage:
+
+- Maintainers identify and fix any bugs or model errors surfaced during
+  the SME Review following the processes outlined in the branch management
+  guidelines for version updates.
+
+- The Scientific Director prepares:
+
+  - A high-level statement of model outcomes
+  - A technical summary of the review process
+
+These summaries are shared with SME Maintainers for input.
+
+
+Release Approval and Publication
+--------------------------------
+
+After SME Maintainers have had an opportunity to review and provide
+feedback on the evaluation report and technical summaries:
+
+1. RuFaS Directors vote on the release:
+
+   - Director of Software Engineering
+   - Director of Strategic Partnerships and Governance
+   - Scientific Director
+
+2. The high-level summary of the SME review is combined with the release
+   note.
+
+3. The technical summary of the SME review is combined with release notes
+   and added to the RuFaS GitHub Pages documentation.
+
+4. The Director of Software Engineering generates the official release.

docs/_src/_wiki/Version-Update-Workflow.rst

@@ -0,0 +1,55 @@
+Maintainer Team Workflow and Communication during a Version Update
+===================================================================
+
+This page describes the process that RuFaS Maintainers will use to communicate
+and execute a version update.
+
+In addition to the updated model and scientific documentation in the repository
+codebase, additional deliverables of this process for **Major** and **Minor**
+updates are:
+
+- A detailed technical report of the scientific review by SME Maintainers
+- A version release note that includes:
+  - A high-level summary of the outcomes of the SME Review
+  - A summary of the changelog
+
+
+Before Merging ``dev`` to ``test``
+------------------------------
+
+**Major** and **Minor** updates:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+1. RuFaS Maintainers meet to discuss what work **will and will not** be
+   included in the release and the proposed timeline.
+
+2. The PML reviews and approves pending work to be included and excluded
+   from the version update, with additional Maintainer input as needed.
+
+3. The PML and team leaders create a GitProject for the version release and
+   add pending issues/PRs that must be completed before the version update
+   (before ``dev → test``). If a Maintainer identifies additional work that would benefit
+   the version release after the project is created, issues/PRs should be added to
+   the version release project or discussed with the PML at the members discretion. 
+
+4. SME Maintainers review the scientific evaluation process and criteria from the
+   previous version release and update as necessary.
+
+
+**Patch** Updates
+^^^^^^^^^^^^^^^^^^
+
+Patch updates do not require a full project. Bug fixes required in the patch should be collected 
+as sub-issues under a **Patch version update parent issue**.
+
+
+After ``test`` is updated
+-------------------------
+- During the testing phase, SME Maintainers will evaluate integrated model performance. 
+(link to Scientific Evaluation Process). 
+- When the model performance has been verified by at least one SME Maintainer per Module, 
+the Scientific Director will prepare a Scientific Report summarizing the model performance and changes in scientific methodology. 
+- The Director of Software Engineering will lead preparation of a set of Release Notes to summarize model updates. 
+- When the testing phase is complete, both the Scientific Report and Release notes are added to the project documentation on GitHub
+and shared with the PML for approval.

docs/_src/index.rst

@@ -68,6 +68,7 @@ Team Culture
 
    _wiki/RuFaS-Versioning-Policy
    _wiki/SCRUM-at-RuFaS
+   _wiki/Version-Update-Workflows
 
 Release Notes
 -------------