Genetics Implementation by allisterakun · Pull Request #2734 · RuminantFarmSystems/RuFaS

allisterakun · 2026-02-05T15:57:24Z

Implements the Genetics submodule into the Animal module, replacing the simple net_merit attribute with a full Genetics class that tracks true breeding values (TBV), estimated breeding values (EBV), phenotypes, and environmental effects for fat and protein traits.

Context

Issue(s) closed by this pull request: closes issue #2511 (a sub-issue under #2614), #2838.
Original PR: #2608. See previous discussion there.

Documentation

Design doc (last updated Dec 2, 2025)
Genetics Technical Solutions (last updated Apr 15, 2026)
Ongoing documentation issue: Scientific Documentation: Animal Genetics Update #2943

What

Integrated the Genetics class into the Animal class, replacing the net_merit float attribute with a full genetics model tracking TBV, EBV, phenotypes, permanent/temporary environmental effects, and a ranking index for fat and protein traits.
Added a GeneticHistory data type and update_genetic_history method on Animal to record genetic state at each simulation day.
Updated AnimalConfig to load average_phenotype and top_listing_semen datasets during initialization.
Updated Animal.__init__ to accept a RufasTime object instead of a simulation_day integer, and to initialize Genetics differently for newborn calves (with optional dam TBV values) vs. existing animals.
Updated MilkProductionStatistics to include genetic attributes (TBV, EBV, phenotype, environmental effects, ranking index).
Updated ReproductionInputs to pass dam_tbv_fat and dam_tbv_protein instead of net_merit.
Removed net_merit from all animal typed dicts (CalfValuesTypedDict, HeiferI/II/IIIValuesTypedDict, CowValuesTypedDict).
Removed animal_net_merit from REQUIRED_FILE_BLOBS in input_manager.py.
Moved and reorganized genetics-related input data files (e.g., NetMerit_HO.csv, TopListingSemen_HO.csv, mean_phenotype.csv) into input/data/animal/animal_genetics/.
Updated metadata JSON files (freestall, open lot, no animal, and e2e testing) to reference the new genetics data file paths.
Updated default.json properties file with genetics-related configuration.
Updated simulation_engine.py, herd_factory.py, herd_manager.py, and animal_module_reporter.py to accommodate the new genetics integration.
Updated unit tests across multiple test files to reflect the new genetics attributes, RufasTime parameter, and removed net_merit references.
Updated util.py with supporting changes.

Why

The previous implementation used a single net_merit float value to represent animal genetics, which was an oversimplification. The new Genetics submodule provides a biologically meaningful representation of animal genetics by modeling true breeding values, estimated breeding values, phenotypes, and environmental effects for milk fat and protein traits. This enables more accurate simulation of genetic progress, selection decisions, and milk production variability within the herd.

How

The Genetics class computes TBV, EBV, phenotypes, and environmental effects based on birth year, animal type, parity, and optional dam genetic values. For newborn calves with known dam TBV values, genetics are initialized using parental inheritance. For all other animals (existing herd or calves without dam data), genetics are initialized based on population-level data from average_phenotype and top_listing_semen datasets loaded into AnimalConfig. The Animal constructor now requires a RufasTime object to determine birth year/month for genetic initialization. Genetic state is tracked over time via GeneticHistory records appended each simulation day. The net_merit field has been fully replaced by the ranking_index computed within the Genetics class.

Test plan

Updated existing unit tests in test_animal.py, test_animal_config.py, test_herd_factory.py, test_herd_manager_daily_routines.py, test_herd_manager_herd_statistics.py, test_animal_module_reporter.py, test_milk_production.py, test_reproduction.py, and test_simulation_engine.py to accommodate the new Genetics integration and RufasTime parameter.
Added unit tests in test_animal_genetics.py for the Genetics class.
Updated end-to-end test metadata files to reference the new genetics input data files.

Input Changes

Added: input/data/animal/animal_genetics/NetMerit_HO.csv, input/data/animal/animal_genetics/TopListingSemen_HO.csv
Deleted: input/data/animal_genetics/TopListingSemen_HO.csv, input/data/animal_genetics/mean_phenotype.csv (moved to new location under input/data/animal/animal_genetics/)
Modified: All metadata JSON files updated to reference new genetics data file paths (freestall_e2e_metadata.json, open_lot_e2e_metadata.json, no_animal_e2e_metadata.json, example_freestall_dairy_metadata.json, example_open_lot_metadata.json, example_no_animal_metadata.json)
Modified: input/metadata/properties/default.json updated with genetics-related properties.

Output Changes

Animal.genetics: New Genetics object replacing the net_merit float attribute.
Animal.genetic_history: New list of GeneticHistory records tracking genetic state over time.
Animal.milk_statistics: Now includes TBV_fat, TBV_protein, E_permanent_fat, E_permanent_protein, E_temporary_fat, E_temporary_protein, phenotype_fat, phenotype_protein, EBV_fat, EBV_protein, ranking_index.
Animal.net_merit: Removed from Animal and all animal typed dicts.

Filter

Genetics History

"AnimalModuleReporter._report_all_animals_genetic_history.*"

Daily Milk

"AnimalModuleReporter.report_milk.*"

Daily Average Genetics (Herd, Calves, HeiferIs, HeiferIIs, HeiferIIIs, Cows)

"AnimalModuleReporter.report_average_genetics.*"

github-actions · 2026-02-05T16:03:42Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1677
Mypy errors on dev branch: 1677
No difference in error counts

This reverts commit 2ab48cd, reversing changes made to 66ac5b1. revert

github-actions · 2026-02-10T21:44:25Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1740
Mypy errors on dev branch: 1682
58 more errors on genetics_implementation branch

github-actions · 2026-02-10T21:44:25Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Flake8 linting errors were found. Please fix the linting issues.

github-actions · 2026-02-11T16:58:24Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1739
Mypy errors on dev branch: 1681
58 more errors on genetics_implementation branch

github-actions · 2026-02-11T16:58:25Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

YijingGong · 2026-02-17T21:14:42Z

Original PR: #2608
See design doc, PR description, and discussion there.

jadamchick · 2026-03-23T20:05:49Z

I just tried to run the branch as-is, and got the following error (pasted below). Looks like input manager is still looking for animal_net_merit in the metadata?

Starting task: 1/1
[23-Mar-2026_Mon_15-31-27.033331][ERROR][freestall] Metadata blobs error. Missing required file blobs: ['animal_net_merit']. Please add all missing file blobs to metadata.
[23-Mar-2026_Mon_15-31-27.045494][ERROR][freestall] Failed to finish task: 1/1 with output prefix: freestall. Failed to recover from error: Missing required file blobs: ['animal_net_merit']; traceback: Traceback (most recent call last):
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\task_manager.py", line 590, in task
    is_data_valid = TaskManager.handle_input_data_audit(args, input_manager, output_manager, True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\task_manager.py", line 753, in handle_input_data_audit
    is_data_valid = input_manager.start_data_processing(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\input_manager.py", line 128, in start_data_processing
    self._validate_required_file_blobs(set(self.__metadata[ADDRESS_TO_INPUTS].keys()))
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\input_manager.py", line 182, in _validate_required_file_blobs
    raise ValueError(f"Missing required file blobs: {list(missing_blobs)}")
ValueError: Missing required file blobs: ['animal_net_merit']

allisterakun · 2026-03-23T22:17:31Z

I just tried to run the branch as-is, and got the following error (pasted below). Looks like input manager is still looking for animal_net_merit in the metadata?

Starting task: 1/1
[23-Mar-2026_Mon_15-31-27.033331][ERROR][freestall] Metadata blobs error. Missing required file blobs: ['animal_net_merit']. Please add all missing file blobs to metadata.
[23-Mar-2026_Mon_15-31-27.045494][ERROR][freestall] Failed to finish task: 1/1 with output prefix: freestall. Failed to recover from error: Missing required file blobs: ['animal_net_merit']; traceback: Traceback (most recent call last):
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\task_manager.py", line 590, in task
    is_data_valid = TaskManager.handle_input_data_audit(args, input_manager, output_manager, True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\task_manager.py", line 753, in handle_input_data_audit
    is_data_valid = input_manager.start_data_processing(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\input_manager.py", line 128, in start_data_processing
    self._validate_required_file_blobs(set(self.__metadata[ADDRESS_TO_INPUTS].keys()))
  File "C:\Users\jms349\GitHub\RuFaS\RUFAS\input_manager.py", line 182, in _validate_required_file_blobs
    raise ValueError(f"Missing required file blobs: {list(missing_blobs)}")
ValueError: Missing required file blobs: ['animal_net_merit']

Should be fixed now!

github-actions · 2026-03-23T22:24:17Z

Current Coverage: %

Mypy errors on genetics_implementation branch: 1269
Mypy errors on dev branch: 1214
55 more errors on genetics_implementation branch

github-actions · 2026-03-23T22:24:17Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Some tests have failed.

…FarmSystems/RuFaS into genetics_implementation

github-actions · 2026-04-16T03:29:23Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1169
Mypy errors on dev branch: 1190
21 fewer errors on genetics_implementation branch

github-actions · 2026-04-16T03:29:24Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

github-actions · 2026-04-16T07:40:52Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1169
Mypy errors on dev branch: 1190
21 fewer errors on genetics_implementation branch

github-actions · 2026-04-16T07:40:53Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

matthew7838

LGTM, left a bunch of comments, but they are minor. Be sure to update e2e filters again before merging, since the herd size management PR was merged yesterday, and that likely changes some outputs. Nice work, Allister.

github-actions · 2026-04-16T15:07:23Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1169
Mypy errors on dev branch: 1190
21 fewer errors on genetics_implementation branch

github-actions · 2026-04-16T15:07:23Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

github-actions · 2026-04-16T16:28:59Z

Current Coverage: 99%

Mypy errors on genetics_implementation branch: 1180
Mypy errors on dev branch: 1190
10 fewer errors on genetics_implementation branch

github-actions · 2026-04-16T16:29:00Z

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

allisterakun and others added 3 commits February 5, 2026 10:56

Update changelog.md

e88fb45

Update changelog.md

66ac5b1

Merge branch 'dev' into genetics_implementation

2ab48cd

allisterakun and others added 7 commits February 10, 2026 16:18

Revert "Merge branch 'dev' into genetics_implementation"

b8536f4

This reverts commit 2ab48cd, reversing changes made to 66ac5b1. revert

2721

1f723ab

manure sync with dev

afd4baf

Merge branch 'dev' into genetics_implementation

f6b50a2

Merge f6b50a2 into 2f73b6a

ef499f2

Apply Black Formatting

3e1c3e0

Update badges on README

36e8c62

allisterakun and others added 4 commits February 11, 2026 11:52

Merge branch 'dev' into genetics_implementation

16486e5

Merge 16486e5 into 6e54aeb

fe775e1

Apply Black Formatting

9e20d2a

Update badges on README

86fb626

allisterakun marked this pull request as ready for review March 3, 2026 17:16

Update input_manager.py

e72c006

allisterakun and others added 4 commits March 23, 2026 18:18

Merge branch 'dev' into genetics_implementation

879fda1

Merge 879fda1 into 19a697b

7be9224

Apply Black Formatting

e67e054

Update badges on README

64c0432

allisterakun requested a review from matthew7838 April 16, 2026 02:55

allisterakun and others added 8 commits April 15, 2026 23:19

Merge branch 'dev' into genetics_implementation

0fb2f7a

Merge 0fb2f7a into 609e7b5

952d574

Apply Black Formatting

ff27c44

Update test_animal.py

e780baa

Merge branch 'genetics_implementation' of https://github.com/Ruminant…

ddef102

…FarmSystems/RuFaS into genetics_implementation

Merge ddef102 into 609e7b5

f4e2915

Apply Black Formatting

795691d

Update badges on README

32550c0

matthew7838 and others added 3 commits April 16, 2026 16:34

Merge branch 'dev' into genetics_implementation

53447fd

Merge 53447fd into 194023d

8c65c32

Apply Black Formatting

b992d40

matthew7838 approved these changes Apr 16, 2026

View reviewed changes

allisterakun and others added 3 commits April 16, 2026 11:00

implement Matthew's comment

222eec1

Merge 222eec1 into 194023d

39194f1

Apply Black Formatting

c1ddec4

allisterakun and others added 4 commits April 16, 2026 12:22

update e2e

e636f11

Merge e636f11 into 194023d

d921073

Apply Black Formatting

52a8769

Update badges on README

6899e48

allisterakun merged commit a1a8638 into dev Apr 16, 2026
1 check passed

allisterakun deleted the genetics_implementation branch April 16, 2026 16:48

Conversation

allisterakun commented Feb 5, 2026 • edited by YijingGong Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Documentation

What

Why

How

Test plan

Input Changes

Output Changes

Filter

Genetics History

Daily Milk

Daily Average Genetics (Herd, Calves, HeiferIs, HeiferIIs, HeiferIIIs, Cows)

Uh oh!

github-actions bot commented Feb 5, 2026

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

github-actions bot commented Feb 10, 2026

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

github-actions bot commented Feb 11, 2026

Uh oh!

YijingGong commented Feb 17, 2026

Uh oh!

jadamchick commented Mar 23, 2026

Uh oh!

allisterakun commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Mar 23, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

matthew7838 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

github-actions bot commented Apr 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

allisterakun commented Feb 5, 2026 •

edited by YijingGong

Loading