diff --git a/predpreygrass/evolutionary/README.md b/predpreygrass/evolutionary/README.md index 8558ee5a0..342ed8638 100644 --- a/predpreygrass/evolutionary/README.md +++ b/predpreygrass/evolutionary/README.md @@ -60,6 +60,13 @@ sustainability numbers. genuine per-individual search decoupled from the shared PPO policy. Sustainability/coexistence solved; selection-driven drift **null (reversed on the headline metric)** after replication — see `RESULTS.md`. +* **[eco_evolutionary_cultural_plasticity](eco_evolutionary_cultural_plasticity)** — + a structurally different mechanism, not another single-scalar trait: gene-culture + coevolution (dual inheritance). A heritable `plasticity` gene sets how readily an + agent's live, socially-transmitted `dialect` tracks the local same-species + majority; dialect-matching at a catch/graze event earns a coordination-game energy + bonus. Implemented and unit-tested; no training pilot or replication run launched + yet — see `RESULTS.md`. See **[RESULTS.md](RESULTS.md)** for the full cross-module trial log — the sequence of attempts, why each pivot happened, and the current state of the search. diff --git a/predpreygrass/evolutionary/RESULTS.md b/predpreygrass/evolutionary/RESULTS.md index ffdae6c4b..9192e1fbf 100644 --- a/predpreygrass/evolutionary/RESULTS.md +++ b/predpreygrass/evolutionary/RESULTS.md @@ -338,6 +338,60 @@ direction) not yet made. --- +## Trial 8 — `eco_evolutionary_cultural_plasticity` — implemented, not yet run + +**Mechanism, not just another trait.** Trials 1-7 all share one structural +property regardless of trait shape: a single heritable channel feeding a +shared per-species PPO policy. This trial adds a **second, independent +inheritance channel** instead — gene-culture coevolution / dual inheritance +(Boyd & Richerson, 1985; Cavalli-Sforza & Feldman, 1981), not a variant of +the same single-channel design tried seven times so far. + +**The two channels:** `dialect` (categorical, one of 4 arbitrary same-species +coordination codes) is a *live, mutable* per-agent state — seeded from a +heritable founder value at birth, but free to change many times within one +agent's own lifetime via social learning from neighbors within +`culture_range`. `plasticity` (continuous, `[0,1]`, standard Gaussian-mutation +genome trait) is the actual gene under test: it sets how readily an agent's +live dialect updates toward the local same-species majority. The gene does +not encode behavior directly — it encodes capacity to adopt culture, which is +the literal Baldwin/dual-inheritance mechanism. + +**Why this targets the diagnosis in the theoretical note below more directly +than Trial 7 did.** A catch/graze event grants an energy bonus when an +agent's live dialect matches its local majority at that moment — a +coordination game, not a gradient. Unlike Trial 7's combinatorial +needle-in-a-haystack (still null, both species, reversed on the headline +metric), which fixes gap #1 (a rugged landscape) but only partially addresses +gap #2 (genuine individual-lifetime search), this design's social-learning +update is frequency-dependent and history-dependent by construction, and the +gene being tested (`plasticity`) has direct, continuously-measurable leverage +over how fast that individual-level process runs — a tighter version of gap +#2 than Trial 7's per-step joint-guess mechanism, which the genome could not +influence at all (only zero-vs-nonzero WRONG-loci viability gated it). + +**Reverse leg, addressed directly instead of left unconfirmed.** Every prior +module kept the genome invisible to the policy's observation space. This +module deliberately breaks that convention (`include_culture_in_obs`): the +policy observes its own live dialect and local-majority-match status, so PPO +can learn to condition movement/hunting on cultural state — a plausible route +to the "genome/culture shift feeds back into learned behavior" leg no prior +trial confirmed. + +**What's ported unchanged:** satiation-throttle sustainability mechanism, +reproduction/energy dynamics, and the `genome_neutral_drift_control` / +Mann-Whitney multi-seed replication methodology — all directly from +`eco_evolutionary_metabolic_code`. `plasticity` uses the |deviation from +founder| test (à la `metabolic_rate`/`investment`), not Trial 7's directional +one, since plasticity has no a-priori predicted drift direction. + +**Status:** implemented, 27 unit tests passing, 300-step random-policy smoke +run clean. No PPO training pilot or replication run launched yet — see +`eco_evolutionary_cultural_plasticity/RESULTS.md` for what remains before a +real Darwin-signal test can be read from this module. + +--- + ## Theoretical note — Hinton & Nowlan (1987), a candidate future trait direction This motivated the Trial 7 pivot above (`eco_evolutionary_metabolic_code`) — recorded here in full since it's the theoretical basis for that module's design, not just a historical note anymore. diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/README.md b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/README.md new file mode 100644 index 000000000..d89796cc9 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/README.md @@ -0,0 +1,150 @@ +# Cultural Plasticity: Gene-Culture Coevolution (Dual Inheritance) + +## Why this module exists + +Every prior module in this family (`eco_evolutionary_metabolic_rate`, +`eco_evolutionary_investment`, `eco_evolutionary_cooperation`, +`eco_evolutionary_metabolic_code`) evolves a **single heritable channel**: a +genome trait feeding a shared per-species PPO policy. All four came back null +on selection-driven drift after proper neutral-drift-control replication (see +`predpreygrass/evolutionary/RESULTS.md`, Trials 1-7) — either because the trait +sits on a smooth fitness landscape ordinary selection can already climb +unassisted (Hinton & Nowlan's diagnosis for Trials 1-6), or, for the one +genuinely rugged trait tried (`metabolic_code`'s combinatorial loci, Trial 7), +still null. + +This module tries a structurally different mechanism instead of another +variation on "one gene, one shared policy": **dual inheritance / gene-culture +coevolution** (Boyd & Richerson, 1985; Cavalli-Sforza & Feldman, 1981). Two +inheritance channels, not one, evolving at different speeds: + +- **Genetic** (slow, vertical-only, mutation-selection): `plasticity`, a + continuous trait inherited through the existing `Genome` machinery. +- **Cultural** (fast, vertical *and* horizontal, imitation-driven): `dialect`, + a live, mutable, per-agent state that can change many times within one + agent's own lifetime via social learning from neighbors. + +Critically, the gene does **not** encode behavior directly — it encodes +*capacity to adopt culture*. This is the literal Baldwin/dual-inheritance +move, and it targets a landscape that is inherently non-smooth (a +coordination game, not a scalar optimum), which is the shape prior modules' +theoretical note said was missing for a Baldwin effect to have anything to +act on. + +## The two channels + +**`dialect`** (categorical, one of `n_dialects` = 4 arbitrary codes, no code +intrinsically better than another): every agent has a *founder* dialect, +inherited like any other genome field, and a separate *live* dialect +(`agent_live_dialect`), seeded from the founder value at birth but free to +change within that agent's own lifetime. Every `plasticity_check_interval` +steps, each agent looks at the live dialects of same-species neighbors within +`culture_range` (Chebyshev distance) and, with probability equal to its own +`plasticity`, adopts the local majority (`_apply_cultural_learning`). This is +a genuinely per-individual lifetime process, deliberately independent of the +shared PPO policy — the same architectural move `eco_evolutionary_metabolic_code` +used for its per-individual loci-solving, but here decoupled at a landscape +that is frequency-dependent rather than a fixed combination lock. + +**`plasticity`** (continuous, `[0, 1]`, Gaussian mutation on reproduction, +same convention as every prior module's scalar trait): the actual Darwinian +gene under test. It sets how readily an agent's culture tracks the local +consensus. High-plasticity lineages track a locally-advantageous dialect +fast; low-plasticity lineages stay put. This is the trait the replication +methodology below tests for selection-driven drift on — not `dialect` itself +(frequency-dependent drift on `dialect` is close to a foregone conclusion and +not informative on its own). + +## Fitness effect: a coordination game, not a gradient + +A catch (predator) or graze (prey) event grants `coordination_bonus_multiplier` +(default 1.5x) on the energy gained *if* the agent's current live dialect +matches its local same-species majority at that moment +(`_dialect_match_bonus`). This is deliberately **not** a smooth function of +any single scalar — whether a given dialect pays off depends entirely on what +neighbors currently do, which itself depends on the history of social +learning in that neighborhood. A dialect that is locally dominant in one +region may be a minority elsewhere. This is the "needle in a haystack" shape +Hinton & Nowlan's own limitation note says a Baldwin effect needs, achieved +through a coordination game instead of a combinatorial lock. + +## The reverse leg: policy conditioned on cultural state + +`include_culture_in_obs` (default `True`) adds two observation channels: the +agent's own live dialect (normalized) and whether it currently matches the +local majority. This lets the shared PPO policy learn to condition +movement/hunting on rapidly-changing cultural state (e.g., clustering with +same-dialect neighbors) — a plausible route to the "genome/culture shift +feeds back into learned behavior" leg that every prior module left +unconfirmed. Every prior module kept the genome fully invisible to the +policy; this module is a deliberate departure from that convention because +the reverse leg specifically requires the policy to be able to *see* the +state genes have leverage over. + +## What's deliberately unchanged + +- **Offspring investment** is a fixed, non-heritable constant + (`offspring_investment_fraction`, default 0.35) — the heritable channels + here are plasticity/dialect, not investment fraction. +- **Sustainability mechanism** (predator satiation cooldown + per-catch + energy cap) is ported unchanged from `eco_evolutionary_metabolic_rate` / + `eco_evolutionary_investment` / `eco_evolutionary_metabolic_code` — an + already-validated, orthogonal concern (criteria 1/2 of the project goal), + not something this trial re-tests. +- **Reproduction/energy dynamics** are otherwise identical to + `eco_evolutionary_metabolic_code`. + +## Predictions and what to watch in training + +1. **Primary (dual-inheritance Darwin-signal) test:** live-population mean + `plasticity` should drift away from the founder mean (0.1) under real + selection, and — critically — should drift **further than the + neutral-drift control** (`genome_neutral_drift_control=True`, same + mechanism ported from every prior module). This is the headline + comparison in `analyze_replication_seeds.py` + (`live_culture/{species}_plasticity_mean`), using the |deviation from + founder| methodology (no a-priori direction, unlike `metabolic_code`'s + directional mean-WRONG-loci test). +2. **Individual-level cross-check:** `{species}_plasticity_repro_spearman` + (genome plasticity vs. binary reproduced-or-not, this episode) — a more + direct test than the population-mean metric, since it doesn't need + selection to move the aggregate before it's detectable. +3. **Secondary (cultural dynamics) signals, not the headline test:** + `{species}_dialect_entropy` (0 = one dialect has fixed locally; `ln(4)` = + maximum diversity) and `{species}_dialect_match_rate` (fraction of + catch/graze events that earned the coordination bonus) — these should + move even under the neutral control (culture is not genetically + inherited in the sense being tested), so they're diagnostic of the + cultural-learning mechanism working at all, not of selection on + `plasticity`. + +Metrics are logged both per-step (`live_culture/*`, currently-alive +population only — `_build_live_culture_metrics`) and per-episode +(`eco_evolution/*`, live + completed agents this episode — +`_build_episode_training_metrics`), mirroring the dual-metric pattern used by +every prior module in this family. + +## Methodology this module inherits + +Directly ported from `eco_evolutionary_metabolic_code` / `eco_evolutionary_investment`: +- **Satiation-throttle sustainability** — validated, not re-tested here. +- **`genome_neutral_drift_control`** — severs genome inheritance from + reproductive success (offspring template becomes a uniformly random + currently-alive same-species agent instead of the true parent); population/ + energy dynamics unchanged. Isolates mutation + finite-population sampling + noise from genuine selection. +- **Pilot-first discipline** — a short single-seed pilot before committing to + a full 3-seed-per-group Mann-Whitney replication. Every prior module in + this project paid for skipping this step at least once. + +## Status + +Implemented and unit-tested (27 tests covering genome sampling/mutation for +both channels, the cultural-learning update rule and its plasticity-gating, +the coordination-bonus energy-gain hook, neutral-drift-control template +selection, and the metrics builders), plus a 300-step random-policy smoke run +with no errors. No PPO training pilot or replication run launched yet — +launching one is a separate, expensive decision, not automatic once +implementation lands. See `predpreygrass/evolutionary/RESULTS.md`'s Trial 8 +entry for the current state of that decision, and `RESULTS.md` in this +directory for the (currently empty) run log. diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/RESULTS.md b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/RESULTS.md new file mode 100644 index 000000000..d8de50898 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/RESULTS.md @@ -0,0 +1,65 @@ +# Training Analysis — eco_evolutionary_cultural_plasticity + +**Status: implementation + verification only. No PPO training pilot or +replication run has been launched yet.** See `README.md` for the trait +design (dual inheritance: `plasticity` genetic, `dialect` cultural) and +`predpreygrass/evolutionary/RESULTS.md`'s Trial 8 entry for the cross-module +framing and the current state of the follow-up decision on whether/when to +launch a real run. + +--- + +## 1. What's been done so far + +- **Implementation.** `predpreygrass_rllib_env.py`, `utils/genome.py`, and + the config pair (`config_env_eco_evolutionary.py` / + `config_env_eco_evolutionary_neutral_control.py`) built by adapting + `eco_evolutionary_metabolic_code`'s scaffold: same reproduction/energy/ + satiation-throttle mechanics, new dual-inheritance genome + (`plasticity` + `dialect`), new `_apply_cultural_learning` / + `_local_majority_dialect` / `_dialect_match_bonus` mechanism, two new + observation channels (`include_culture_in_obs`), and `live_culture/*` / + `eco_evolution/*` metrics (`_build_live_culture_metrics`, + `_build_episode_training_metrics`). +- **Unit tests.** 27 tests in `tests/test_eco_evolutionary_validation.py`, + all passing: genome founder sampling and bounds for both the continuous + (`plasticity`) and categorical (`dialect`) fields; zero- and full-rate + mutation behavior for both fields; the cultural-learning update rule + (check-interval gating, plasticity-gated adoption, zero-plasticity never + adopts); `_local_majority_dialect` correctness (excluding the focal + agent's own vote); the coordination-bonus energy-gain hook on both match + and mismatch; `genome_neutral_drift_control` template selection; the + RLlib multi-agent contract tests (termination/truncation/observation + bookkeeping) ported unchanged from `eco_evolutionary_metabolic_code`; + dialect-entropy and the dependency-free Spearman helper (including the + tie-averaging fix needed because one of its two inputs, reproduced-or-not, + is binary and ties heavily). +- **Smoke runs.** 300 steps under a uniform-random policy (real config) and + 100 steps under the neutral-drift-control config, both to completion with + no errors, plausible-looking `plasticity`/`dialect_entropy`/ + `dialect_match_rate` metrics at episode end. This is not a training run — + no learning occurred — it only exercises the full step/reproduction/ + cultural-learning/metrics code path at realistic population scale. + +## 2. Not yet done + +- **Sustainability/coexistence pilot** under PPO training (criteria 1/2 of + the project's Darwin/Baldwin goal). The satiation-throttle constants are + ported unchanged from already-validated modules, so this is expected to + transfer, but hasn't been confirmed for this specific environment variant + (the two new observation channels and the coordination-bonus energy + dynamics are new). +- **Real vs. neutral-control multi-seed replication** on `plasticity` drift + (criterion 3 — the actual dual-inheritance selection test). Requires the + pilot above first, per this project's own established discipline (every + prior module paid for skipping straight to replication at least once). +- **Parameter tuning.** `n_dialects=4`, `culture_range=3`, + `coordination_bonus_multiplier=1.5`, `plasticity_check_interval=5`, and + the founder `plasticity_mean=0.1` are reasonable starting defaults, not + validated against real training data. + +## 3. Next step + +Launch a short single-seed pilot (a few hundred iterations) to check +sustainability/coexistence before committing to a full replication — +same pilot-first discipline used by every prior module in this family. diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/__init__.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/analyze_replication_seeds.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/analyze_replication_seeds.py new file mode 100644 index 000000000..e4abc8680 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/analyze_replication_seeds.py @@ -0,0 +1,159 @@ +""" +Trial 8 analysis: aggregate multi-seed replication runs (real config vs. +neutral-drift control) and compare `plasticity` drift magnitude between the +two groups -- same real-vs-control replication methodology used for +metabolic_rate/investment (|deviation from founder|, Mann-Whitney U, +one-sided "real > control"), applied to this module's dual-inheritance trait. + +`plasticity` has no a-priori directional prediction (unlike +eco_evolutionary_metabolic_code's mean_wrong_loci, where selection should push +the value down) -- founder mean is 0.1 with no assumed sign of drift, so this +mirrors metabolic_rate/investment's |deviation from founder| approach rather +than metabolic_code's directional one. + +Expects experiment directories under ~/ray_results/ named: + PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_SEED_ (real) + PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_NEUTRAL_CONTROL_SEED_ (control) + +as produced by tune_ppo_cultural_plasticity.py / tune_ppo_cultural_plasticity_neutral_control.py +when run with --seed. Missing runs are reported, not treated as an error, so this +can be run before all seeds have finished. + +Caveat printed alongside every result: with only ~3 runs per group, this is a +directional check, not a well-powered significance test -- Mann-Whitney U with +n=3 vs n=3 cannot reach conventional significance thresholds even in the best +case. Treat the p-values as a rough indicator, not proof either way. + +Usage: + python predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/analyze_replication_seeds.py +""" + +import json +import re +from pathlib import Path + +import numpy as np +from scipy.stats import mannwhitneyu + +RAY_RESULTS_DIR = Path("~/ray_results").expanduser() +FOUNDER_MEAN = 0.1 # see founder_genome.{predator,prey}.plasticity_mean in config_env_eco_evolutionary.py + +REAL_PATTERN = re.compile(r"^PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_SEED(\d+)_") +CONTROL_PATTERN = re.compile(r"^PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_NEUTRAL_CONTROL_SEED(\d+)_") + + +def find_seed_runs(pattern: re.Pattern) -> dict[int, Path]: + """Return {seed: result.json path} for the most recent run of each seed.""" + matches: dict[int, tuple[float, Path]] = {} + for exp_dir in RAY_RESULTS_DIR.glob("PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY*"): + m = pattern.match(exp_dir.name) + if not m: + continue + seed = int(m.group(1)) + result_jsons = list(exp_dir.glob("*/result.json")) + if not result_jsons: + continue + result_json = result_jsons[0] + mtime = result_json.stat().st_mtime + if seed not in matches or mtime > matches[seed][0]: + matches[seed] = (mtime, result_json) + return {seed: path for seed, (_, path) in matches.items()} + + +def load_series(result_json: Path, key: str) -> list[float]: + values = [] + with open(result_json) as f: + for line in f: + d = json.loads(line) + er = d.get("env_runners", {}) or {} + v = er.get(key) + if v is not None and not (isinstance(v, float) and np.isnan(v)): + values.append(float(v)) + return values + + +def summarize_run(result_json: Path) -> dict: + """Q1/Q5 mean and max |deviation from founder| for predator and prey live plasticity.""" + out = {} + for species in ("predator", "prey"): + series = load_series(result_json, f"live_culture/{species}_plasticity_mean") + if not series: + out[species] = None + continue + n = len(series) + q = max(n // 5, 1) + q1_mean = float(np.mean(series[:q])) + q5_mean = float(np.mean(series[-q:])) + max_abs_dev = float(np.max(np.abs(np.array(series) - FOUNDER_MEAN))) + out[species] = { + "n_points": n, + "q1_mean": q1_mean, + "q5_mean": q5_mean, + "net_change": q5_mean - q1_mean, + "final_dev_from_founder": q5_mean - FOUNDER_MEAN, + "max_abs_dev_from_founder": max_abs_dev, + } + return out + + +def main(): + real_runs = find_seed_runs(REAL_PATTERN) + control_runs = find_seed_runs(CONTROL_PATTERN) + + print("=== Runs found ===") + print(f"Real: seeds {sorted(real_runs)} ({len(real_runs)} found)") + print(f"Neutral control: seeds {sorted(control_runs)} ({len(control_runs)} found)") + print() + + real_summaries = {seed: summarize_run(p) for seed, p in sorted(real_runs.items())} + control_summaries = {seed: summarize_run(p) for seed, p in sorted(control_runs.items())} + + print("=== Per-run summary (plasticity) ===") + header = f"{'group':<8}{'seed':<6}{'species':<10}{'n':<6}{'Q1':<8}{'Q5':<8}{'net_chg':<9}{'|dev_final|':<12}{'max|dev|':<9}" + print(header) + for group_name, summaries in (("real", real_summaries), ("control", control_summaries)): + for seed, s in summaries.items(): + for species in ("predator", "prey"): + r = s.get(species) + if r is None: + print(f"{group_name:<8}{seed:<6}{species:<10} no data") + continue + print( + f"{group_name:<8}{seed:<6}{species:<10}{r['n_points']:<6}" + f"{r['q1_mean']:<8.4f}{r['q5_mean']:<8.4f}{r['net_change']:<+9.4f}" + f"{abs(r['final_dev_from_founder']):<12.4f}{r['max_abs_dev_from_founder']:<9.4f}" + ) + print() + + print("=== Mann-Whitney U: real vs. control plasticity-drift magnitude ===") + print("CAVEAT: n=3 vs n=3 (or fewer) has very limited power -- cannot reach") + print("conventional significance thresholds even in the best case. Read the") + print("direction and effect size, not the p-value alone.\n") + + for species in ("predator", "prey"): + for metric_name, metric_key in ( + ("final |deviation from founder|", "final_dev_from_founder"), + ("max |deviation from founder|", "max_abs_dev_from_founder"), + ): + real_vals = [ + abs(s[species][metric_key]) if metric_key == "final_dev_from_founder" else s[species][metric_key] + for s in real_summaries.values() if s.get(species) is not None + ] + control_vals = [ + abs(s[species][metric_key]) if metric_key == "final_dev_from_founder" else s[species][metric_key] + for s in control_summaries.values() if s.get(species) is not None + ] + if len(real_vals) < 2 or len(control_vals) < 2: + print(f"{species:<10} {metric_name:<32} not enough runs yet " + f"(real n={len(real_vals)}, control n={len(control_vals)})") + continue + stat, p = mannwhitneyu(real_vals, control_vals, alternative="greater") + print( + f"{species:<10} {metric_name:<32} real={np.mean(real_vals):.4f} " + f"(n={len(real_vals)}) control={np.mean(control_vals):.4f} (n={len(control_vals)}) " + f"U={stat:.1f} p(real>control)={p:.3f}" + ) + + +if __name__ == "__main__": + main() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary.py new file mode 100644 index 000000000..bdc0438f7 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary.py @@ -0,0 +1,102 @@ +config_env = { + "seed": 41, + "max_steps": 1000, + # Grid and Observation Settings + "grid_size": 25, + # Observation channels: predators, prey, grass. Grid edges are handled by + # clipping the observation window and leaving out-of-grid cells at zero. + "num_obs_channels": 3, + "predator_obs_range": 7, + "prey_obs_range": 9, + # Action space settings: 3x3 Moore neighbourhood (8 directions + stay). + "action_range": 3, + # Rewards + "reproduction_reward_predator": { + "predator": 10.0, + }, + "reproduction_reward_prey": { + "prey": 10.0, + }, + # Energy settings + "energy_loss_per_step_predator": 0.15, + "energy_loss_per_step_prey": 0.05, + "movement_energy_cost_per_cell_predator": 0.0, + "movement_energy_cost_per_cell_prey": 0.0, + "predator_creation_energy_threshold": 12.0, + "prey_creation_energy_threshold": 8.0, + "initial_energy_predator_at_reset": 5.0, + "initial_energy_prey_at_reset": 3.0, + # Individual-level throttles on predator hunting (satiation), ported + # unchanged from eco_evolutionary_metabolic_rate/investment/metabolic_code + # -- an already-validated, orthogonal sustainability mechanism (criteria + # 1/2), not something this trial (Trial 8) is re-testing. Regulates + # predator population growth through each predator's own recent hunting + # success (a Holling-type handling-time mechanism) rather than an + # artificial population-level rule. Steps after a catch before the same + # predator can catch again ("digesting"). + "predator_satiation_cooldown": 8, + # Per-catch energy cap ("satiation ceiling") -- a predator can't extract + # more than this from a single kill regardless of the prey's own energy. + "max_energy_gain_per_prey": 8.0, + # Offspring investment is a FIXED, non-heritable constant in this module + # (see README.md "What's deliberately unchanged") -- the heritable + # channels here are plasticity/dialect (see below), not investment + # fraction. + "offspring_investment_fraction": 0.35, + # --- Dual-inheritance genome: two heritable channels --- + # `plasticity` (continuous, [0,1]): the Darwinian trait -- the per-check + # probability an agent's live dialect updates toward the local majority. + # `dialect` (categorical, one of n_dialects): the founder cultural bias a + # newborn is seeded with; freely overridable within its own lifetime. + "genome_enabled": True, + "n_dialects": 4, + "founder_genome": { + "predator": {"plasticity_mean": 0.1, "plasticity_std": 0.05}, + "prey": {"plasticity_mean": 0.1, "plasticity_std": 0.05}, + }, + # Per-reproduction-event mutation of the continuous plasticity trait + # (Gaussian perturbation, same convention as every prior module's scalar + # trait mutation). + "genome_mutation": { + "rate": 0.05, + "std": 0.03, + }, + # Per-reproduction-event mutation of the categorical dialect trait + # (uniform resample among all n_dialects, same convention as + # eco_evolutionary_metabolic_code's per-locus mutation). + "dialect_mutation": { + "rate": 0.05, + }, + "trait_bounds": { + "plasticity": (0.0, 1.0), + }, + # --- Cultural mechanics (not heritable; see predpreygrass_rllib_env.py) --- + # Chebyshev-distance radius defining a same-species "local culture group" + # for both the social-learning check and the coordination-bonus lookup. + "culture_range": 3, + # Energy-gain multiplier applied to a catch/graze event when the agent's + # live dialect matches its local majority at that moment. + "coordination_bonus_multiplier": 1.5, + # Steps between per-agent social-learning checks. + "plasticity_check_interval": 5, + # Expose own live dialect + local-majority-match flag as two extra + # observation channels, so the shared PPO policy can condition + # movement/hunting on cultural state (the "reverse leg"). + "include_culture_in_obs": True, + # Absolute energy caps + "max_energy_grass": 2.0, + # Learning agents + "n_possible_predators": 200, + "n_possible_prey": 1000, + "n_initial_active_predators": 6, + "n_initial_active_prey": 8, + # Grass settings + "initial_num_grass": 100, + "initial_energy_grass": 2.0, + "energy_gain_per_step_grass": 0.04, + "verbose_engagement": False, + "verbose_movement": False, + "verbose_decay": False, + "verbose_reproduction": False, + "debug_mode": False, +} diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary_neutral_control.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary_neutral_control.py new file mode 100644 index 000000000..866b3b299 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_env_eco_evolutionary_neutral_control.py @@ -0,0 +1,10 @@ +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env as _base_config_env + +# Neutral-drift control: identical to the real experiment config in every respect +# except genome_neutral_drift_control, so any difference in genome-trait drift +# (plasticity in particular) between this run and a real run is attributable to +# selection, not to some other config difference. See RESULTS.md. +config_env = { + **_base_config_env, + "genome_neutral_drift_control": True, +} diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_cpu_eco_evolutionary.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_cpu_eco_evolutionary.py new file mode 100644 index 000000000..df0f0da60 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_cpu_eco_evolutionary.py @@ -0,0 +1,25 @@ +config_ppo = { + "max_iters": 250, + # Core learning + "lr": 5e-5, + "gamma": 0.99, + "lambda_": 0.95, + "train_batch_size_per_learner": 1024, + "minibatch_size": 128, + "num_epochs": 6, + "entropy_coeff": 0.01, + "vf_loss_coeff": 1.0, + "clip_param": 0.3, + # Resources + "num_learners": 1, + "num_env_runners": 6, + "num_envs_per_env_runner": 1, + "num_gpus_per_learner": 0, + "num_cpus_for_main_process": 1, + "num_cpus_per_env_runner": 1, + "sample_timeout_s": 600, + "rollout_fragment_length": "auto", + # KL / exploration + "kl_coeff": 0.2, + "kl_target": 0.01, +} diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_gpu_eco_evolutionary.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_gpu_eco_evolutionary.py new file mode 100644 index 000000000..1a21702a6 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/config/config_ppo_gpu_eco_evolutionary.py @@ -0,0 +1,25 @@ +config_ppo = { + "max_iters": 1000, + # Core learning + "lr": 0.0003, + "gamma": 0.99, + "lambda_": 1.0, + "train_batch_size_per_learner": 1024, # 2048, + "minibatch_size": 128, # 256, + "num_epochs": 10, # 15, + "entropy_coeff": 0.0, + "vf_loss_coeff": 1.0, + "clip_param": 0.3, + # Resources + "num_learners": 1, + "num_env_runners": 7, + "num_envs_per_env_runner": 3, + "num_gpus_per_learner": 1, + "num_cpus_for_main_process": 4, + "num_cpus_per_env_runner": 4, + "sample_timeout_s": 600, + "rollout_fragment_length": "auto", + # KL / exploration + "kl_coeff": 0.2, + "kl_target": 0.01, +} diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_debug.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_debug.py new file mode 100644 index 000000000..ff17c071a --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_debug.py @@ -0,0 +1,583 @@ +""" +This script loads (pre) trained PPO policy modules (RLModules) directly from a checkpoint +and runs them in the PredPreyGrass eco-evolutionary environment for interactive debugging. + +This version differs from ppg_2_policies in that it includes two types of predators and two types of prey, +making distinct behaviors and characteristics possible per species. In this version, the "speed 2" +version of predator and prey are are faster and can cover more ground in one movement step. +Both speed 1 and speed 2 predators and prey are mutually trained. Evaluation of only speed 1 +predators and prey with only small change of mutation to speed 2 predators and prey generally +leads to dominance of speed 2 agents and extinction of speed 1 agents as the trained model shows +in the simulation. + +The simulation can be controlled in real-time using a graphical interface. +- [Space] Pause/Unpause +- [->] Step Forward +- [<-] Step Backward +- Tooltips are available to inspect agent IDs, positions, energies. + +The environment is rendered using PyGame, and the simulation can be recorded as a video. +""" +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass # Import the custom environment +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.matplot_renderer import CombinedEvolutionVisualizer, PreyDeathCauseVisualizer +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.pygame_grid_renderer_rllib import PyGameRenderer, ViewerControlHelper, LoopControlHelper + +# external libraries +import ray +from ray.rllib.core.rl_module.rl_module import RLModule +from ray.tune.registry import register_env +import torch +from datetime import datetime +import os +import json +import pygame +import cv2 +import numpy as np +import re +from collections import defaultdict +from typing import Callable, cast + + +SAVE_EVAL_RESULTS = True +SAVE_MOVIE = False +MOVIE_FILENAME = "simulation.mp4" +MOVIE_FPS = 10 + +RAY_RESULTS_DIR = os.path.expanduser("~/ray_results") +CHECKPOINT_ROOT = ( + "PPO_ECO_EVOLUTION_2026-06-23_10-30-31/" + "PPO_PredPreyGrass_cc36a_00000_0_2026-06-23_10-30-31" +) + + +def env_creator(config): + return PredPreyGrass(config) + + +def cv2_video_writer_fourcc(*codec: str) -> int: + video_writer_fourcc = cast(Callable[..., int], getattr(cv2, "VideoWriter_fourcc")) + return video_writer_fourcc(*codec) + + +def _latest_checkpoint_path(run_path): + checkpoints = [] + for name in os.listdir(run_path): + path = os.path.join(run_path, name) + if os.path.isdir(path) and re.fullmatch(r"checkpoint_\d+", name): + checkpoints.append((int(name.rsplit("_", 1)[1]), path)) + if not checkpoints: + raise FileNotFoundError(f"No checkpoint directories found in: {run_path}") + return max(checkpoints, key=lambda item: item[0])[1] + + +def _load_training_env_config_from_run(checkpoint_path, base_cfg): + """ + Try to locate a run_config.json near the checkpoint and merge its env settings + into the provided base_cfg. This aligns evaluation observations with the + shapes the policy network was trained on (avoids matmul shape errors). + """ + candidates = [ + os.path.join(os.path.dirname(checkpoint_path), "run_config.json"), + os.path.join(os.path.dirname(os.path.dirname(checkpoint_path)), "run_config.json"), + ] + training_env_cfg = None + for cand in candidates: + if os.path.isfile(cand): + try: + with open(cand, "r") as f: + rc = json.load(f) + # Prefer explicit env_config if present; else, fall back to top-level keys + if isinstance(rc, dict): + if isinstance(rc.get("env_config"), dict): + training_env_cfg = rc["env_config"] + elif isinstance(rc.get("config_env"), dict): + training_env_cfg = rc["config_env"] + else: + # Heuristic: intersect with known keys in base_cfg + training_env_cfg = {k: rc[k] for k in base_cfg.keys() if k in rc} + break + except Exception: + pass + + if not isinstance(training_env_cfg, dict): + return base_cfg # nothing to merge + + # Only override observation-critical keys. + obs_keys = { + "grid_size", + "num_obs_channels", + "predator_obs_range", + "prey_obs_range", + # Action ranges can affect obs encoding in some setups; include for safety + "action_range", + "include_speed_in_obs", + "include_move_available_in_obs", + } + merged = dict(base_cfg) + for k in obs_keys: + if k in training_env_cfg: + merged[k] = training_env_cfg[k] + return merged + + +def policy_mapping_fn(agent_id, *args, **kwargs): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unrecognized agent_id format: {agent_id}") + + +def policy_pi(observation, policy_module, deterministic=True): + obs_tensor = torch.tensor(observation).float().unsqueeze(0) + with torch.no_grad(): + action_output = policy_module._forward_inference({"obs": obs_tensor}) + logits = action_output.get("action_dist_inputs") + if logits is None: + raise KeyError("policy_pi: action_dist_inputs not found in action_output.") + if deterministic: + return torch.argmax(logits, dim=-1).item() + else: + dist = torch.distributions.Categorical(logits=logits) + return dist.sample().item() + + +def setup_environment_and_visualizer(now): + run_path = os.path.join(RAY_RESULTS_DIR, CHECKPOINT_ROOT) + checkpoint_path = _latest_checkpoint_path(run_path) + checkpoint_dir = os.path.basename(checkpoint_path) + print(f"Loading checkpoint: {checkpoint_path}") + # training_dir = os.path.dirname(checkpoint_path) + eval_output_dir = os.path.join(checkpoint_path, f"eval_{checkpoint_dir}_{now}") + + rl_module_dir = os.path.join(checkpoint_path, "learner_group", "learner", "rl_module") + module_paths = {} + + if os.path.isdir(rl_module_dir): + for pid in os.listdir(rl_module_dir): + path = os.path.join(rl_module_dir, pid) + if os.path.isdir(path): + module_paths[pid] = path + else: + raise FileNotFoundError(f"RLModule directory not found: {rl_module_dir}") + + rl_modules = {pid: RLModule.from_checkpoint(path) for pid, path in module_paths.items()} + + # Build config based on config_env and align observation-related keys with training run config + cfg = dict(config_env) + cfg = _load_training_env_config_from_run(checkpoint_path, cfg) + + env = env_creator(config=cfg) + grid_size = (env.grid_size, env.grid_size) + # Try to configure FOV overlay similar to random policy, but tolerate legacy renderer + try: + visualizer = PyGameRenderer( + grid_size, + cell_size=32, + enable_speed_slider=True, + enable_tooltips=True, + max_steps=cfg.get("max_steps", 1000), + predator_obs_range=cfg.get("predator_obs_range"), + prey_obs_range=cfg.get("prey_obs_range"), + show_fov=True, + fov_alpha=40, + fov_agents=["predator_0", "prey_0"], + ) + except TypeError: + visualizer = PyGameRenderer( + grid_size, + cell_size=32, + enable_speed_slider=True, + enable_tooltips=True, + max_steps=cfg.get("max_steps", 1000), + ) + + if SAVE_EVAL_RESULTS: + os.makedirs(eval_output_dir, exist_ok=True) + with open(os.path.join(eval_output_dir, "config_env.json"), "w") as f: + json.dump(cfg, f, indent=4) + ceviz = CombinedEvolutionVisualizer(destination_path=eval_output_dir, timestamp=now) + pdviz = PreyDeathCauseVisualizer(destination_path=eval_output_dir, timestamp=now) + else: + ceviz = CombinedEvolutionVisualizer(destination_path=None, timestamp=now) + pdviz = PreyDeathCauseVisualizer(destination_path=None, timestamp=now) + + return env, visualizer, rl_modules, ceviz, pdviz, eval_output_dir + + +def step_backwards_if_requested( + control, env, snapshots, time_steps, predator_counts, prey_counts, energy_by_type_series, visualizer, ceviz, pdviz +): + if control.step_backward: + if len(snapshots) > 1: + snapshots.pop() + env.restore_state_snapshot(snapshots[-1]) + print(f"[ViewerControl] Step Backward → Step {env.current_step}") + observations = {agent: env._get_observation(agent) for agent in env.agents} + if time_steps: + time_steps.pop() + predator_counts.pop() + prey_counts.pop() + energy_by_type_series.pop() + + ceviz.record(agent_ids=env.agents) + pdviz.record(env.death_cause_prey) + visualizer.update( + grass_positions=env.grass_positions, + grass_energies=env.grass_energies, + step=env.current_step, + agents_just_ate=env.agents_just_ate, + per_step_agent_data=env.per_step_agent_data, + ) + + control.fps_slider_rect = visualizer.slider_rect + pygame.time.wait(100) + control.step_backward = False + return observations + control.step_backward = False + return None + +def step_forward( + env, + observations, + rl_modules, + control, + visualizer, + ceviz, + pdviz, + snapshots, + predator_counts, + prey_counts, + time_steps, + energy_by_type_series, + total_reward, + clock, + SAVE_MOVIE, + video_writer, +): + action_dict = {} + for agent_id in env.agents: + group = policy_mapping_fn(agent_id) + if group in rl_modules: + action_dict[agent_id] = policy_pi(observations[agent_id], rl_modules[group], deterministic=True) + + observations, rewards, terminations, truncations, _ = env.step(action_dict) + # print("----------------------------------------------") + # print(f"Step {env.current_step}") + # print(f"Rewards: {rewards}") + # print(f"Terminations: {terminations}") + # print("Terminated agents:", {k: v for k, v in terminations.items() if v is True}) + # print("----------------------------------------------") + + snapshots.append(env.get_state_snapshot()) + if len(snapshots) > 100: + snapshots.pop(0) + + ceviz.record(agent_ids=env.agents) + if hasattr(ceviz, "record_energy"): + ceviz.record_energy(env.get_total_energy_by_type()) + + visualizer.update( + grass_positions=env.grass_positions, + grass_energies=env.grass_energies, + step=env.current_step, + agents_just_ate=env.agents_just_ate, + per_step_agent_data=env.per_step_agent_data, + dead_prey=getattr(env, "dead_prey", None), + ) + control.fps_slider_rect = visualizer.slider_rect + + if SAVE_MOVIE: + frame = pygame.surfarray.array3d(visualizer.screen) + frame = np.transpose(frame, (1, 0, 2)) + frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) + video_writer.write(frame) + + control.step_once = False + clock.tick(visualizer.target_fps) + + predator_counts.append(sum(1 for a in env.agents if "predator" in a)) + prey_counts.append(sum(1 for a in env.agents if "prey" in a)) + time_steps.append(env.current_step) + energy_by_type_series.append(env.get_total_energy_by_type()) + total_reward += sum(rewards.values()) + + return observations, total_reward, terminations, truncations + +def render_static_if_paused(env, visualizer): + visualizer.update( + grass_positions=env.grass_positions, + grass_energies=env.grass_energies, + step=env.current_step, + agents_just_ate=env.agents_just_ate, + per_step_agent_data=env.per_step_agent_data, + dead_prey=getattr(env, "dead_prey", None), + ) + +def parse_uid(uid): + """ + Parse agent id like 'predator_2#17' into sortable components: + -> ('predator', 2, 17) + """ + match = re.match(r"((?:predator|prey))_(\d+)(?:#(\d+))?", uid) + if match: + group, idx, lifetime = match.groups() + return group, int(idx), int(lifetime) if lifetime is not None else 0 + else: + return uid, float("inf"), float("inf") # fallback for malformed ids + +def print_ranked_reward_summary(env, total_reward): + def _get_group_rewards(env): + group_rewards = defaultdict(list) + for uid, stats in env.get_all_agent_stats().items(): + reward = stats.get("cumulative_reward", 0.0) + group, index, reuse = parse_uid(uid) + group_rewards[group].append((uid, reward, index, reuse)) + return group_rewards + + def _format_ranked_reward_summary(env, total_reward, group_rewards): + lines = [] + lines.append(f"Total Reward: {total_reward:.2f}\n") + lines.append("--- Ranked Reward Breakdown per Unique Agent ---\n") + for group in sorted(group_rewards.keys()): + lines.append(f"\n## {group.replace('_', ' ').title()} ##\n") + sorted_group = sorted( + group_rewards[group], key=lambda x: (-x[1], x[2], x[3]) + ) + for uid, reward, _, _ in sorted_group: + lines.append(f"{uid:25}: {reward:.2f}\n") + lines.append("\n--- Aggregated Totals ---\n") + lines.append(f"Total number of steps: {env.current_step - 1}\n") + for group in sorted(group_rewards.keys()): + total = sum(r for _, r, _, _ in group_rewards[group]) + lines.append(f"Total {group.replace('_', ' ').title():25}: {total:.2f}\n") + lines.append(f"Total All-Agent Reward: {total_reward:.2f}\n") + return lines + + group_rewards = _get_group_rewards(env) + lines = _format_ranked_reward_summary(env, total_reward, group_rewards) + print("\nEvaluation complete! Total Reward: {:.2f}".format(total_reward)) + for line in lines[1:]: # skip duplicate first line + print(line.rstrip()) + +def save_reward_summary_to_file(env, total_reward, output_dir): + reward_log_path = os.path.join(output_dir, "reward_summary.txt") + def _get_group_stats(env): + group_stats = defaultdict(list) + for uid, stats in env.get_all_agent_stats().items(): + group, index, reuse = parse_uid(uid) + reward = stats.get("cumulative_reward", 0.0) + lifetime = (stats.get("death_step") or env.current_step) - stats.get("birth_step", 0) + offspring = stats.get("offspring_count", 0) + off_per_step = offspring / lifetime if lifetime > 0 else 0.0 + group_stats[group].append({ + "uid": uid, + "reward": reward, + "lifetime": lifetime, + "offspring": offspring, + "off_per_step": off_per_step, + "index": index, + "reuse": reuse, + }) + return group_stats + + def _format_ranked_fitness_summary(env, total_reward, group_stats): + lines = [] + lines.append(f"Total Reward: {total_reward:.2f}\n") + lines.append("--- Ranked Reward Breakdown per Unique Agent ---\n") + for group in sorted(group_stats.keys()): + lines.append(f"\n## {group.replace('_', ' ').title()} ##\n") + sorted_group = sorted( + group_stats[group], key=lambda x: (-x["reward"], x["index"], x["reuse"]) + ) + lines.append( + f"{'Agent':25} | {'R':>8} | {'Life':>6} | {'Off':>4} | {'Off/100':>8}\n" + ) + for entry in sorted_group: + lines.append( + f"{entry['uid']:25} | " + f"{entry['reward']:8.2f} | " + f"{entry['lifetime']:6} | " + f"{entry['offspring']:4} | " + f"{100*entry['off_per_step']:8.2f}\n" + ) + # Print averages for this group + n = len(sorted_group) + if n > 0: + avg_reward = sum(e['reward'] for e in sorted_group) / n + avg_life = sum(e['lifetime'] for e in sorted_group) / n + avg_offspring = sum(e['offspring'] for e in sorted_group) / n + avg_off_per_step = sum(e['off_per_step'] for e in sorted_group) / n + lines.append( + f"{'(Averages)':25} | {avg_reward:8.2f} | {avg_life:6.1f} | {avg_offspring:4.2f} | " + f"{100*avg_off_per_step:8.2f}\n" + ) + lines.append("\n--- Aggregated Totals ---\n") + lines.append(f"Total number of steps: {env.current_step - 1}\n") + for group in sorted(group_stats.keys()): + total = sum(e['reward'] for e in group_stats[group]) + lines.append(f"Total {group.replace('_', ' ').title():25}: {total:.2f}\n") + lines.append(f"Total All-Agent Reward: {total_reward:.2f}\n") + return lines + + group_stats = _get_group_stats(env) + lines = _format_ranked_fitness_summary(env, total_reward, group_stats) + with open(reward_log_path, "w") as f: + for line in lines: + f.write(line) + +def run_post_evaluation_plots(ceviz, pdviz): + if SAVE_EVAL_RESULTS: + ceviz.plot() + pdviz.plot() + +def print_ranked_fitness_summary(env): + print("\n--- Ranked Fitness Summary by Group ---") + group_stats = defaultdict(list) + for uid, stats in env.get_all_agent_stats().items(): + group, index, reuse = parse_uid(uid) + lifetime = (stats["death_step"] or env.current_step) - stats["birth_step"] + group_stats[group].append( + { + "uid": uid, + "reward": stats.get("cumulative_reward", 0.0), + "lifetime": lifetime, + "offspring": stats.get("offspring_count", 0), + "off_per_step": stats.get("offspring_count", 0) / lifetime if lifetime > 0 else 0.0, + } + ) + + for group in sorted(group_stats.keys()): + print(f"\n## {group.replace('_', ' ').title()} ##") + sorted_group = sorted(group_stats[group], key=lambda x: (-x["offspring"], -x["reward"], -x["lifetime"])) + print( + f"{'Agent':25} | {'R':>8} | {'Life':>6} | {'Off':>4} | {'Off/100':>8}" + ) + for entry in sorted_group[:10]: # top 10 + print( + f"{entry['uid']:25} | " + f"{entry['reward']:8.2f} | " + f"{entry['lifetime']:6} | " + f"{entry['offspring']:4} | " + f"{100*entry['off_per_step']:8.2f}" + ) + # Print averages for this group + n = len(sorted_group) + if n > 0: + avg_reward = sum(e['reward'] for e in sorted_group) / n + avg_life = sum(e['lifetime'] for e in sorted_group) / n + avg_offspring = sum(e['offspring'] for e in sorted_group) / n + avg_off_per_step = sum(e['off_per_step'] for e in sorted_group) / n + print( + f"{'(Averages)':25} | {avg_reward:8.2f} | {avg_life:6.1f} | {avg_offspring:4.2f} | " + f"{100*avg_off_per_step:8.2f}" + ) + +if __name__ == "__main__": + seed = 4 + ray.init(ignore_reinit_error=True) + now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + register_env("PredPreyGrass", lambda config: env_creator(config)) + + env, visualizer, rl_modules, ceviz, pdviz, eval_output_dir = setup_environment_and_visualizer(now) + observations, _ = env.reset(seed=seed) + + if SAVE_MOVIE: + screen_width = visualizer.screen.get_width() + screen_height = visualizer.screen.get_height() + fourcc = cv2_video_writer_fourcc(*"mp4v") + video_writer = cv2.VideoWriter(MOVIE_FILENAME, fourcc, MOVIE_FPS, (screen_width, screen_height)) + else: + video_writer = None + + control = ViewerControlHelper() + loop_helper = LoopControlHelper() + control.fps_slider_rect = visualizer.slider_rect + control.fps_slider_update_fn = lambda new_fps: setattr(visualizer, "target_fps", new_fps) + control.visualizer = visualizer + + clock = pygame.time.Clock() + total_reward = 0 + predator_counts, prey_counts, time_steps = [], [], [] + snapshots = [env.get_state_snapshot()] + energy_by_type_series = [env.get_total_energy_by_type()] + + while not loop_helper.simulation_terminated: + control.handle_events() + + new_obs = step_backwards_if_requested( + control, env, snapshots, time_steps, predator_counts, prey_counts, energy_by_type_series, visualizer, ceviz, pdviz + ) + if new_obs is not None: + observations = new_obs + continue + + if loop_helper.should_step(control): + observations, total_reward, terminations, truncations = step_forward( + env, + observations, + rl_modules, + control, + visualizer, + ceviz, + pdviz, + snapshots, + predator_counts, + prey_counts, + time_steps, + energy_by_type_series, + total_reward, + clock, + SAVE_MOVIE, + video_writer, + ) + loop_helper.update_simulation_terminated(terminations, truncations) + else: + render_static_if_paused(env, visualizer) + pygame.time.wait(50) + + # === Print total offspring by type === + offspring_counts = env.get_total_offspring_by_type() + print("\n--- Offspring Counts by Type ---") + for agent_type, count in offspring_counts.items(): + print(f"{agent_type:20}: {count}") + + # print_ranked_reward_summary(env, total_reward) + + # print("Death statistics:", env.death_agents_stats) + + if SAVE_EVAL_RESULTS: + save_reward_summary_to_file(env, total_reward, eval_output_dir) + energy_log_path = os.path.join(eval_output_dir, "energy_by_type.json") + with open(energy_log_path, "w") as f: + json.dump(energy_by_type_series, f, indent=2) + # Export per-agent event log for offline analysis + def _convert(obj): + if isinstance(obj, (int, float, str)) or obj is None: + return obj + if isinstance(obj, dict): + return {k: _convert(v) for k, v in obj.items()} + if isinstance(obj, (list, tuple)): + return [_convert(v) for v in obj] + try: + return obj.item() + except Exception: + return str(obj) + + event_log_path = os.path.join(eval_output_dir, f"agent_event_log_{now}.json") + with open(event_log_path, "w", encoding="utf-8") as f: + json.dump({aid: _convert(rec) for aid, rec in env.agent_event_log.items()}, f, indent=2) + print(f"Agent event log written to: {event_log_path}") + # Always show plots on screen + ceviz.plot() + if SAVE_EVAL_RESULTS: + # Export all unique agent fitness stats + agent_fitness_path = os.path.join(eval_output_dir, "agent_fitness_stats.json") + with open(agent_fitness_path, "w") as f: + json.dump(env.get_all_agent_stats(), f, indent=2) + + print_ranked_fitness_summary(env) + + pygame.quit() + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_default.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_default.py new file mode 100644 index 000000000..f25ff9ae8 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_default.py @@ -0,0 +1,126 @@ +import os +from datetime import datetime +import ray +import torch +import pygame +import cv2 +import numpy as np + +from ray.rllib.core.rl_module.rl_module import RLModule +from ray.tune.registry import register_env + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.pygame_grid_renderer_rllib import PyGameRenderer + +# ==== CONFIG ==== +RAY_RESULTS_DIR = "/home/doesburg/Projects/PredPreyGrass/predpreygrass/rllib_termination/experiments/" +CHECKPOINT_PATH = "legacy/checkpoints_2025_11_01/checkpoint_000004" + +SAVE_MOVIE = False +MOVIE_FILENAME = "eval_video.mp4" +MOVIE_FPS = 10 +SEED = 5 +# ================ + + +def policy_mapping_fn(agent_id): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unrecognized agent_id format: {agent_id}") + + +def policy_pi(observation, policy_module, deterministic=True): + obs_tensor = torch.tensor(observation).float().unsqueeze(0) + with torch.no_grad(): + output = policy_module._forward_inference({"obs": obs_tensor}) + logits = output.get("action_dist_inputs") + return torch.argmax(logits, dim=-1).item() if deterministic else torch.distributions.Categorical(logits=logits).sample().item() + + +def load_rl_modules(checkpoint_path): + rl_module_dir = os.path.join(checkpoint_path, "learner_group", "learner", "rl_module") + module_paths = { + pid: os.path.join(rl_module_dir, pid) + for pid in os.listdir(rl_module_dir) + if os.path.isdir(os.path.join(rl_module_dir, pid)) + } + return {pid: RLModule.from_checkpoint(path) for pid, path in module_paths.items()} + + +if __name__ == "__main__": + ray.init(ignore_reinit_error=True) + register_env("PredPreyGrass", lambda config: PredPreyGrass(config)) + now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + + checkpoint_root = os.path.join(RAY_RESULTS_DIR, CHECKPOINT_PATH) + rl_modules = load_rl_modules(checkpoint_root) + + env = PredPreyGrass(config=config_env) + grid_size = (env.grid_size, env.grid_size) + visualizer = PyGameRenderer( + grid_size, + cell_size=32, + enable_speed_slider=False, + enable_tooltips=False, + show_fov=True, + fov_alpha=40, + fov_agents=["predator_0", "prey_0"], + predator_obs_range=config_env.get("predator_obs_range", 7), + prey_obs_range=config_env.get("prey_obs_range", 9), + ) + + observations, _ = env.reset(seed=SEED) + # FOV overlays will be shown for: predator_0 and prey_0 + + if SAVE_MOVIE: + screen_size = visualizer.screen.get_size() + fourcc = cv2.VideoWriter_fourcc(*"mp4v") + video_writer = cv2.VideoWriter(MOVIE_FILENAME, fourcc, MOVIE_FPS, screen_size) + else: + video_writer = None + + total_reward = 0 + clock = pygame.time.Clock() + + terminated, truncated = False, False + while not terminated and not truncated: + action_dict = {aid: policy_pi(observations[aid], rl_modules[policy_mapping_fn(aid)]) for aid in env.agents} + observations, rewards, terminations, truncations, _ = env.step(action_dict) + + visualizer.update( + grass_positions=env.grass_positions, + grass_energies=env.grass_energies, + agents_just_ate=env.agents_just_ate, + step=env.current_step, + per_step_agent_data=env.per_step_agent_data, + ) + + if SAVE_MOVIE: + frame = pygame.surfarray.array3d(visualizer.screen) + frame = np.transpose(frame, (1, 0, 2)) # Pygame to OpenCV format + frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR) + video_writer.write(frame) + + total_reward += sum(rewards.values()) + terminated = all(terminations.values()) + truncated = all(truncations.values()) + clock.tick(visualizer.target_fps) + + print(f"\n Evaluation complete! Total Reward: {total_reward:.2f}") + print(f"Total Steps: {env.current_step}") + + if video_writer: + video_writer.release() + + # Export per-agent event log for offline analysis + out_dir = os.path.join(checkpoint_root, "eval_logs") + os.makedirs(out_dir, exist_ok=True) + event_log_path = os.path.join(out_dir, f"agent_event_log_{now}.json") + env.export_agent_event_log(event_log_path) + print(f"Agent event log written to: {event_log_path}") + + pygame.quit() + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_multi_runs.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_multi_runs.py new file mode 100644 index 000000000..3ac98e0e3 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/evaluate_ppo_from_checkpoint_multi_runs.py @@ -0,0 +1,108 @@ +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.matplot_renderer import CombinedEvolutionVisualizer + +# external libraries +import os +import json +from datetime import datetime +import ray +import torch +from ray.rllib.core.rl_module.rl_module import RLModule +from ray.tune.registry import register_env + + +SAVE_EVAL_RESULTS = True +N_RUNS = 10 # Number of evaluation runs +SEED = 1 + + +def policy_mapping_fn(agent_id): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Invalid agent_id format: {agent_id}") + + +def policy_pi(observation, policy_module, deterministic=True): + obs_tensor = torch.tensor(observation).float().unsqueeze(0) + with torch.no_grad(): + action_output = policy_module._forward_inference({"obs": obs_tensor}) + logits = action_output.get("action_dist_inputs") + if logits is None: + raise KeyError("Missing 'action_dist_inputs' in output.") + return torch.argmax(logits, dim=-1).item() if deterministic else torch.distributions.Categorical(logits=logits).sample().item() + + +def setup_modules(): + ray_results_dir = "/home/doesburg/Dropbox/02_marl_results/predpreygrass_results/ray_results/" + checkpoint_path = "v2_7_tune_default_benchmark/PPO_PredPreyGrass_86337_00000_0_2025-08-04_23-53-58/" + checkpoint_dir = "checkpoint_000099" + checkpoint_root = os.path.abspath(ray_results_dir + checkpoint_path + checkpoint_dir) + rl_module_dir = os.path.join(checkpoint_root, "learner_group", "learner", "rl_module") + module_paths = { + pid: os.path.join(rl_module_dir, pid) + for pid in os.listdir(rl_module_dir) + if os.path.isdir(os.path.join(rl_module_dir, pid)) + } + rl_modules = {pid: RLModule.from_checkpoint(path) for pid, path in module_paths.items()} + return rl_modules, checkpoint_root + + +if __name__ == "__main__": + ray.init(ignore_reinit_error=True) + register_env("PredPreyGrass", lambda config: PredPreyGrass(config)) + now = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + + for run in range(N_RUNS): + seed = SEED + run + print(f"\n=== Evaluation Run {run + 1} / {N_RUNS} ===") + print(f"Using seed: {seed}") + rl_modules, checkpoint_root = setup_modules() + env = PredPreyGrass(config=config_env) + observations, _ = env.reset(seed=SEED + run) # Use different seed per run + if SAVE_EVAL_RESULTS: + eval_output_dir = os.path.join(checkpoint_root, f"eval_runs_{now}") + os.makedirs(eval_output_dir, exist_ok=True) + visualizer = CombinedEvolutionVisualizer(destination_path=eval_output_dir, timestamp=now, run_nr=run + 1) + else: + visualizer = None + + total_reward = 0 + terminated = False + truncated = False + + while not terminated and not truncated: + action_dict = {aid: policy_pi(observations[aid], rl_modules[policy_mapping_fn(aid)]) for aid in env.agents} + observations, rewards, terminations, truncations, _ = env.step(action_dict) + if visualizer: + visualizer.record( + agent_ids=env.agents, + ) + + total_reward += sum(rewards.values()) + # print(f"Step {i} Total Reward so far: {total_reward:.2f}") + terminated = any(terminations.values()) + truncated = any(truncations.values()) + + print(f"Evaluation complete! Total Reward: {total_reward:.2f}") + print(f"Total Steps: {env.current_step}") + # for aid, r in env.cumulative_rewards.items(): + # print(f"{aid:20}: {r:.2f}") + + if SAVE_EVAL_RESULTS: + visualizer.plot() + config_env_dir = os.path.join(eval_output_dir, "config_env") + os.makedirs(config_env_dir, exist_ok=True) + summary_data_dir = os.path.join(eval_output_dir, "summary_data") + os.makedirs(summary_data_dir, exist_ok=True) + with open(os.path.join(config_env_dir, "config_env_" + str(run + 1) + ".json"), "w") as f: + json.dump(config_env, f, indent=4) + with open(os.path.join(summary_data_dir, "reward_summary_" + str(run + 1) + ".txt"), "w") as f: + f.write(f"Total Reward: {total_reward:.2f}\n") + for aid, stats in env.get_all_agent_stats().items(): + r = stats.get("cumulative_reward", 0.0) + f.write(f"{aid:20}: {r:.2f}\n") + + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/predpreygrass_rllib_env.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/predpreygrass_rllib_env.py new file mode 100644 index 000000000..f919c680a --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/predpreygrass_rllib_env.py @@ -0,0 +1,1679 @@ +""" +Predator-Prey Grass RLlib Environment +""" +# external libraries (Ray required) +import gymnasium +from ray.rllib.env.multi_agent_env import MultiAgentEnv +import numpy as np + +from typing import Optional + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.genome import ( + Genome, + GENOME_FIELD_DEFAULTS, + founder_genome, + mutate_genome, +) + + +class PredPreyGrass(MultiAgentEnv): + def __init__(self, config=None): + super().__init__() + if config is None: + raise ValueError("Environment config must be provided explicitly.") + self.config = config + self._initialize_from_config() + + # RLlib necessities in constructor + self.possible_agents = self._build_possible_agent_ids() + + self.observation_spaces = {agent_id: self._build_observation_space(agent_id) for agent_id in self.possible_agents} + + self.action_spaces = {agent_id: self._build_action_space(agent_id) for agent_id in self.possible_agents} + self.observation_space = gymnasium.spaces.Dict({str(aid): space for aid, space in self.observation_spaces.items()}) + self.action_space = gymnasium.spaces.Dict({str(aid): space for aid, space in self.action_spaces.items()}) + self.observation_space_struct = self.observation_spaces + self.action_space_struct = self.action_spaces + + def _initialize_from_config(self): + config = self.config + self.debug_mode = config["debug_mode"] + self.verbose_movement = config["verbose_movement"] + self.verbose_decay = config["verbose_decay"] + self.verbose_reproduction = config["verbose_reproduction"] + self.verbose_engagement = config["verbose_engagement"] + + self.max_steps = config["max_steps"] + # RNG will be initialized during reset to ensure per-episode reproducibility + + # Rewards dictionaries + self.reproduction_reward_predator_config = config["reproduction_reward_predator"] + self.reproduction_reward_prey_config = config["reproduction_reward_prey"] + + # Energy settings + self.energy_loss_per_step_predator = config["energy_loss_per_step_predator"] + self.energy_loss_per_step_prey = config["energy_loss_per_step_prey"] + self.movement_energy_cost_per_cell_predator = config.get("movement_energy_cost_per_cell_predator", 0.0) + self.movement_energy_cost_per_cell_prey = config.get("movement_energy_cost_per_cell_prey", 0.0) + self.predator_creation_energy_threshold = config["predator_creation_energy_threshold"] + self.prey_creation_energy_threshold = config["prey_creation_energy_threshold"] + # Individual-level throttle on predator hunting (satiation), ported from + # eco_evolutionary_metabolic_rate / eco_evolutionary_investment / + # eco_evolutionary_metabolic_code: regulates predator population growth + # through each predator's own recent hunting success (a Holling-type + # handling-time mechanism) rather than a population-level rule. + # 0 disables the cooldown. + self.predator_satiation_cooldown = int(config.get("predator_satiation_cooldown", 0)) + self.max_energy_gain_per_prey = float(config.get("max_energy_gain_per_prey", float("inf"))) + self.max_energy_grass = self.config["max_energy_grass"] + + # Offspring investment is a fixed, non-heritable constant in this module + # -- the heritable channels here are plasticity/dialect (see below), not + # investment fraction. One heritable trait *pair* per module, matching + # every prior module's isolation discipline. + self.offspring_investment_fraction = float(config.get("offspring_investment_fraction", 0.35)) + + # Learning agents + self.n_possible_predators = config["n_possible_predators"] + self.n_possible_prey = config["n_possible_prey"] + + self.n_initial_active_predators = config["n_initial_active_predators"] + self.n_initial_active_prey = config["n_initial_active_prey"] + # Default: ~33% of max, so the policy encounters sparse cold-start conditions. + # Override via config "n_initial_active_predators_min" / "n_initial_active_prey_min" if needed. + self.n_initial_active_predators_min = config.get( + "n_initial_active_predators_min", max(1, self.n_initial_active_predators // 3) + ) + self.n_initial_active_prey_min = config.get( + "n_initial_active_prey_min", max(1, self.n_initial_active_prey // 3) + ) + + self.initial_energy_predator_at_reset = config["initial_energy_predator_at_reset"] + self.initial_energy_prey_at_reset = config["initial_energy_prey_at_reset"] + + # Grid and Observation Settings + self.grid_size = config["grid_size"] + self.num_obs_channels = config["num_obs_channels"] + self.predator_obs_range = config["predator_obs_range"] + self.prey_obs_range = config["prey_obs_range"] + # When True, two extra observation channels expose the agent's own + # live dialect and whether it currently matches the local majority -- + # this is what lets the shared PPO policy condition movement/hunting + # behavior on rapidly-changing cultural state (the "reverse leg" of + # the dual-inheritance loop). See README.md. + self.include_culture_in_obs = bool(config.get("include_culture_in_obs", True)) + + # Grass settings + self.initial_num_grass = config["initial_num_grass"] + self.initial_energy_grass = config["initial_energy_grass"] + self.energy_gain_per_step_grass = config["energy_gain_per_step_grass"] + + # Action range and movement mapping + self.action_range = config["action_range"] + self.genome_enabled = config.get("genome_enabled", True) + # Neutral-drift null model: when True, an offspring's genome template is a + # uniformly random currently-alive same-species agent instead of the agent + # that actually reproduced. Reproduction eligibility, timing, and all energy + # dynamics are unchanged -- only which genome propagates is decoupled from + # reproductive success. Isolates mutation + finite-population sampling noise + # from genuine selection. Ported from eco_evolutionary_investment / + # eco_evolutionary_metabolic_rate / eco_evolutionary_metabolic_code. See + # RESULTS.md. + self.genome_neutral_drift_control = bool(config.get("genome_neutral_drift_control", False)) + self.n_dialects = int(config.get("n_dialects", 4)) + self.genome_config = { + "n_dialects": self.n_dialects, + "founder_genome": config.get("founder_genome", {}), + "genome_mutation": config.get("genome_mutation", {}), + "dialect_mutation": config.get("dialect_mutation", {}), + "trait_bounds": config.get("trait_bounds", {}), + } + # Dual-inheritance cultural mechanics: same-species agents within + # culture_range (Chebyshev distance) of one another form a local + # "culture group". A coordination-game bonus multiplies energy gain + # when an agent's current live dialect matches that group's majority + # dialect at the moment of a catch/graze event -- a direct, one-step + # fitness link, and a rugged, frequency-dependent landscape (unlike + # every smooth-scalar trait tried in Trials 1-6). See README.md. + self.culture_range = int(config.get("culture_range", 3)) + self.coordination_bonus_multiplier = float(config.get("coordination_bonus_multiplier", 1.5)) + # Steps between per-agent social-learning checks (the genuinely + # per-individual lifetime-search channel the plasticity gene has + # leverage over -- deliberately independent of the shared PPO policy, + # same architectural move as eco_evolutionary_metabolic_code's + # per-individual haystack solving). + self.plasticity_check_interval = int(config.get("plasticity_check_interval", 5)) + + def _init_reset_variables(self, seed): + # Agent tracking + self.current_step = 0 + # Seed RNG for this episode: prefer provided reset seed, else fallback to config seed, else default + if seed is None: + seed = self.config["seed"] + self.rng = np.random.default_rng(seed) + + self.agent_positions = {} + self.predator_positions = {} + self.prey_positions = {} + self.grass_positions = {} + self.agent_energies = {} + self.grass_energies = {} + # Per-step return dicts + self.observations, self.rewards, self.terminations, self.truncations, self.infos = {}, {}, {}, {}, {} + # Ages (in steps) for all currently active agents + self.agent_ages = {} + # Step at which each predator's satiation cooldown from its last catch + # ends; a predator on cooldown does not hunt (mirrors digestion/rest). + self.agent_satiation_until = {} + self.satiation_blocked_catches_predator = 0 + # Per-agent event log for detailed post-hoc analysis + # Structure per agent_id: + # { + # "agent_id": str, + # "birth_step": int, + # "death_step": Optional[int], + # "parent_id": Optional[str], + # "death_cause": Optional[str], + # "eating_events": [ + # {"t": int, "id_eaten": str, "energy_after": float}, ...], + # "reproduction_events": [ + # {"t": int, "child_id": str}, ...], + # "reward_events": [ + # {"t": int, "reproduction_reward": float, + # "cumulative_reward": float}, ...], + # "lifecycle_events": [], + # } + self.agent_event_log = {} + self.agent_stats_live = {} + self.agent_stats_completed = {} + self.per_step_agent_data = [] # One entry per step; each is {agent_id: {position, energy, ...}} + self._per_agent_step_deltas = {} # Internal temp storage to track energy deltas during step + self._next_lifetime_id = 0 + self.agent_parents = {} + self.agent_genomes = {} + self.agent_offspring_counts = {} + self.agent_live_offspring_ids = {} + # Cultural-learning live state. Not heritable directly -- seeded from + # the genome's `dialect` field at birth (agent_genomes[...].dialect), + # then evolves independently within that agent's own lifetime via + # _apply_cultural_learning, gated by the heritable `plasticity` trait. + self.agent_live_dialect: dict[str, int] = {} + # Dialect-match bookkeeping for the eco-evolution metrics (episode-level). + self.dialect_match_events_predator = 0 + self.dialect_total_events_predator = 0 + self.dialect_match_events_prey = 0 + self.dialect_total_events_prey = 0 + # IDs ever registered this episode — blocks any reuse within an episode. + # RLlib hard constraint: once termination=True is returned for an ID, that ID + # cannot reappear in subsequent steps. + self.used_agent_ids: set = set() + # Capacity block counters (episode-level) + self.reproduction_blocked_due_to_capacity_predator = 0 + self.reproduction_blocked_due_to_capacity_prey = 0 + # Episode-level spawn counters + self.spawned_predators = 0 + self.spawned_prey = 0 + # Peak simultaneous active counts + self.peak_active_predators = 0 + self.peak_active_prey = 0 + + self._last_live_culture_metrics: dict = {} + self.agents_just_ate = set() + + # Per-step infos accumulator and last-move diagnostics + self._pending_infos = {} + self._last_move_block_reason = {} + + self.death_cause_prey = {} + + self.agent_last_reproduction = {} + + # aggregates per step + self.active_num_predators = 0 + self.active_num_prey = 0 + + n_pred_min = min(self.n_initial_active_predators_min, self.n_initial_active_predators) + n_prey_min = min(self.n_initial_active_prey_min, self.n_initial_active_prey) + n_initial_predators = int(self.rng.integers(n_pred_min, self.n_initial_active_predators + 1)) + n_initial_prey = int(self.rng.integers(n_prey_min, self.n_initial_active_prey + 1)) + + self.agents = [] + for i in range(n_initial_predators): + agent_id = f"predator_{i}" + self.agents.append(agent_id) + self._register_new_agent(agent_id, is_founder=True) + for i in range(n_initial_prey): + agent_id = f"prey_{i}" + self.agents.append(agent_id) + self._register_new_agent(agent_id, is_founder=True) + + self.grass_agents = [f"grass_{i}" for i in range(self.initial_num_grass)] + + self.grid_world_state_shape = (self.num_obs_channels, self.grid_size, self.grid_size) + self.initial_grid_world_state = np.zeros(self.grid_world_state_shape, dtype=np.float32) + self.grid_world_state = self.initial_grid_world_state.copy() + + def _generate_action_map(range_size: int): + delta = (range_size - 1) // 2 + return { + i: (dx, dy) + for i, (dx, dy) in enumerate((dx, dy) for dx in range(-delta, delta + 1) for dy in range(-delta, delta + 1)) + } + + self.action_to_move_tuple_agents = _generate_action_map(self.action_range) + + self._printed_termination_ids = set() + + def _alloc_new_id(self, species: str) -> Optional[str]: + """Return the lowest-index fresh agent ID, or None if pool is exhausted. + + Scans possible_agents in index order and returns the first ID that is not + currently active and not yet used this episode (no reuse within an episode). + """ + prefix = species + "_" + used = self.used_agent_ids + for aid in self.possible_agents: + if str(aid).startswith(prefix) and aid not in self.agents and aid not in used: + return str(aid) + return None + + def reset(self, *, seed=None, options=None): + """ + Reset the environment to its initial state. + """ + super().reset(seed=seed) + self._init_reset_variables(seed) + + self._create_and_place_grid_world_entities() + + self.active_num_predators = len(self.predator_positions) + self.active_num_prey = len(self.prey_positions) + self.current_num_grass = len(self.grass_positions) + + self.current_step = 0 + + self.observation_space_struct = self.observation_spaces + self.action_space_struct = self.action_spaces + observations = {agent: self._get_observation(agent) for agent in self.agents} + return observations, {} + + def step(self, action_dict): + acted_ids = {str(agent) for agent in action_dict} + self.observations, self.rewards, self.terminations, self.truncations, self.infos = {}, {}, {}, {}, {} + self.agents_just_ate.clear() # For stepwise display eating in grid + self._pending_infos = {} # Reset per-step self.infos + + # Step 1: Process energy depletion due to time steps and update age + self._apply_time_step_update() + + # Step 1b: Per-individual social-learning check (see + # _apply_cultural_learning). Runs before this step's engagements so an + # agent that just adopted the local majority dialect already earns the + # coordination bonus on this same step's catch/graze. This is the + # genuinely per-individual lifetime-search channel the plasticity gene + # has leverage over, deliberately independent of the shared PPO policy + # that governs movement/hunting behavior. + self._apply_cultural_learning() + + # Step 2: Regenerate grass energy + self._regenerate_grass_energy() + + # Step 3: process agent movements + self._process_agent_movements(action_dict) + + # Step 4: Handle agent engagements (optimized scans with snapshots) + # 4a) Starvation first: snapshot of energies to avoid repeated dict lookups + energies = self.agent_energies + handle_starv = self._handle_energy_starvation + for agent, energy in tuple(energies.items()): + if energy <= 0: + handle_starv(agent) + + # 4b) Prey engagements over active prey only (skip terminated) + prey_snapshot = tuple(self.prey_positions.keys()) + for agent in prey_snapshot: + if self.terminations.get(agent): + continue + self._handle_prey_engagement(agent) + + # 4c) Predator engagements over active predators only (skip terminated) + predator_snapshot = tuple(self.predator_positions.keys()) + for agent in predator_snapshot: + if self.terminations.get(agent): + continue + self._handle_predator_engagement(agent) + + # Step 5: Handle agent removals + # Collect all agents marked terminated and still present in the active maps + terminations = self.terminations + agent_positions = self.agent_positions + to_remove = [a for a, t in terminations.items() if t and a in agent_positions] + + if to_remove: + to_remove_set = set(to_remove) + energies = self.agent_energies + predator_pos = self.predator_positions + prey_pos = self.prey_positions + + for agent in to_remove: + agent_positions.pop(agent, None) + energies.pop(agent, None) + predator_pos.pop(agent, None) + prey_pos.pop(agent, None) + self.agent_live_dialect.pop(agent, None) + + # Rebuild active agent list without repeated O(n) list.remove calls + self.agents = [a for a in self.agents if a not in to_remove_set] + + # Step 6: Spawning of new agents (cooldown and chance removed; energy-only) + # Use snapshots of active predators/prey to avoid iterating over newly spawned agents in this step + predator_snapshot = tuple(self.predator_positions.keys()) + prey_snapshot = tuple(self.prey_positions.keys()) + energies = self.agent_energies + pred_thr = self.predator_creation_energy_threshold + prey_thr = self.prey_creation_energy_threshold + + for agent in predator_snapshot: + if energies[agent] >= pred_thr: + self._handle_predator_reproduction(agent) + + for agent in prey_snapshot: + if energies[agent] >= prey_thr: + self._handle_prey_reproduction(agent) + + # Step 7: Assemble return dicts. Observations contain only agents that + # should act next. Rewards/done flags may include agents that ended on + # this step. + live_ids: set[str] = {str(a) for a in self.agents} + reward_full = dict(self.rewards) + term_full = dict(self.terminations) + trunc_full = dict(self.truncations) + ended_ids: set[str] = {str(aid) for aid, flag in term_full.items() if flag} | {str(aid) for aid, flag in trunc_full.items() if flag} + + missing_acted_ids = (acted_ids - live_ids - ended_ids) & set(self.possible_agents) + for agent in missing_acted_ids: + # RLlib requires every agent that acted and disappeared to receive a + # final done flag plus final observation + # edge cases where lower-level bookkeeping removed an acted agent + # before the final return dict was assembled. + reward_full.setdefault(agent, self.rewards.get(agent, 0.0)) + term_full[agent] = True + trunc_full[agent] = False + ended_ids |= missing_acted_ids + + episode_done = self.active_num_prey <= 0 or self.active_num_predators <= 0 + if episode_done: + # Extinction is a natural terminal condition, not a time-limit truncation. + for agent in live_ids: + term_full[agent] = True + trunc_full[agent] = False + ended_ids |= live_ids + self._attach_final_cumulative_rewards_for_live_agents() + self._attach_episode_training_metrics() + + output_ids = live_ids | ended_ids + self.rewards = {aid: reward_full.get(aid, 0.0) for aid in output_ids} + self.terminations = {aid: term_full.get(aid, False) for aid in output_ids} + self.truncations = {aid: trunc_full.get(aid, False) for aid in output_ids} + self.infos = {aid: self._pending_infos.get(aid, {}) for aid in output_ids} + + if "__all__" in self._pending_infos: + self.infos["__all__"] = self._pending_infos["__all__"] + self.terminations["__all__"] = episode_done + self.truncations["__all__"] = False + + next_actor_ids = live_ids - ended_ids if not episode_done else set() + get_obs = self._get_observation + final_observations = {} + for agent in ended_ids: + if agent in self.observations: + final_observations[agent] = self.observations[agent] + elif agent in self.agent_positions: + final_observations[agent] = get_obs(agent) + elif agent in acted_ids: + final_observations[agent] = self._empty_observation(agent) + self.observations = final_observations + self.observations.update({agent: get_obs(agent) for agent in next_actor_ids}) + + step_data = {} + for agent in self.agents: + pos = self.agent_positions[agent] + energy = self.agent_energies[agent] + deltas = self._per_agent_step_deltas.get(agent, {"decay": 0.0, "move": 0.0, "eat": 0.0, "repro": 0.0}) + parent = self.agent_parents.get(agent) + age = self.agent_ages[agent] + step_data[agent] = { + "position": pos, + "energy": energy, + "energy_decay": deltas["decay"], + "energy_movement": deltas["move"], + "energy_eating": deltas["eat"], + "energy_reproduction": deltas["repro"], + "age": age, + "offspring_count": self.agent_offspring_counts[agent], + "offspring_ids": self.agent_live_offspring_ids.get(agent, []), + "parent": parent, + } + + self.per_step_agent_data.append(step_data) + self._per_agent_step_deltas.clear() + + self.current_step += 1 + + if self.current_step >= self.max_steps and not episode_done: + # Final time-limit step: active agents are truncated. Agents that + # died earlier in this step remain terminated. RLlib requires a + # final observation for truncated agents for value bootstrapping. + active_now: set[str] = {str(a) for a in self.agents} + final_ids = (set(self.rewards) | set(self.terminations) | set(self.truncations) | active_now) - {"__all__"} + final_ids |= (acted_ids - active_now) & set(self.possible_agents) + + obs, rews, terms, truncs, infos = {}, {}, {}, {}, {} + for agent in active_now: + truncs[agent] = True + terms[agent] = False + obs[agent] = self._get_observation(agent) + + for agent in final_ids: + rews[agent] = self.rewards.get(agent, 0.0) + terms[agent] = self.terminations.get(agent, terms.get(agent, False)) + truncs[agent] = self.truncations.get(agent, truncs.get(agent, False)) + if agent in active_now: + terms[agent] = False + truncs[agent] = True + elif agent in acted_ids and not terms[agent] and not truncs[agent]: + terms[agent] = True + info = self._pending_infos.get(agent, {}).copy() + record = self.agent_stats_live.get(agent) or self.agent_stats_completed.get(agent) + if record is not None: + info["final_cumulative_reward"] = record.get("cumulative_reward", 0.0) + infos[agent] = info + if (terms[agent] or truncs[agent]) and agent not in obs: + if agent in self.observations: + obs[agent] = self.observations[agent] + elif agent in self.agent_positions: + obs[agent] = self._get_observation(agent) + elif agent in acted_ids: + obs[agent] = self._empty_observation(agent) + + truncs["__all__"] = True + terms["__all__"] = False + + # Overwrite step-assembled dicts to only contain final-step relevant agents. + self.observations, self.rewards = obs, rews + self.terminations, self.truncations, self.infos = terms, truncs, infos + + for agent_id in list(self.agent_stats_live.keys()): + self._finalize_agent_record(agent_id, cause="time_limit") + + infos["__all__"] = {"training_metrics": self._build_episode_training_metrics()} + self.agents = [] + + return self.observations, self.rewards, self.terminations, self.truncations, self.infos + + if episode_done: + self.agents = [] + + n_pred = sum(1 for a in self.agents if "predator" in str(a)) + n_prey = sum(1 for a in self.agents if "prey" in str(a)) + if n_pred > self.peak_active_predators: + self.peak_active_predators = n_pred + if n_prey > self.peak_active_prey: + self.peak_active_prey = n_prey + + self._last_live_culture_metrics = self._build_live_culture_metrics() + return self.observations, self.rewards, self.terminations, self.truncations, self.infos + + def _apply_cultural_learning(self): + """Per-check social-learning step: the individual-level lifetime-search + channel this module's genetic trait (plasticity) has leverage over. + + Every `plasticity_check_interval` steps, each living agent samples the + live dialects of same-species neighbors within `culture_range` + (Chebyshev distance) and, with probability equal to its own heritable + `plasticity`, adopts the local majority dialect. Deliberately + decoupled from the shared PPO policy: policy governs movement/ + hunting, this governs which cultural convention the agent currently + follows. This is the second, culturally-transmitted inheritance + channel, distinct from the founder/parent `dialect` value stored in + the (Darwinian) genome. + """ + if self.plasticity_check_interval <= 0 or self.current_step % self.plasticity_check_interval != 0: + return + # Snapshot live dialects before any updates this round so every + # agent's majority calculation this round is simultaneous, not + # sequential (order-independent within one check). + dialect_snapshot = dict(self.agent_live_dialect) + if not dialect_snapshot: + return + positions = self.agent_positions + r = self.culture_range + for agent in self.agents: + agent = str(agent) + genome = self.agent_genomes.get(agent) + if genome is None or agent not in dialect_snapshot: + continue + same_species = "predator" if "predator" in agent else "prey" + x0, y0 = positions[agent] + neighbor_dialects = [ + other_dialect + for other, other_dialect in dialect_snapshot.items() + if other != agent + and same_species in other + and other in positions + and max(abs(positions[other][0] - x0), abs(positions[other][1] - y0)) <= r + ] + if not neighbor_dialects: + continue + counts = np.bincount(neighbor_dialects, minlength=self.n_dialects) + majority_dialect = int(np.argmax(counts)) + if majority_dialect == self.agent_live_dialect[agent]: + continue + if self.rng.random() < float(genome.plasticity): + self.agent_live_dialect[agent] = majority_dialect + + def _local_majority_dialect(self, agent: str) -> Optional[int]: + """Majority live dialect among same-species agents within culture_range.""" + agent = str(agent) + if agent not in self.agent_positions: + return None + x0, y0 = self.agent_positions[agent] + same_species = "predator" if "predator" in agent else "prey" + r = self.culture_range + neighbor_dialects = [ + other_dialect + for other, other_dialect in self.agent_live_dialect.items() + if other != agent + and same_species in other + and other in self.agent_positions + and max( + abs(self.agent_positions[other][0] - x0), + abs(self.agent_positions[other][1] - y0), + ) + <= r + ] + if not neighbor_dialects: + return None + counts = np.bincount(neighbor_dialects, minlength=self.n_dialects) + return int(np.argmax(counts)) + + def _dialect_match_bonus(self, agent: str) -> float: + """Coordination-game multiplier: >1.0 if agent's live dialect matches + its local same-species majority at this moment, else 1.0. Also + updates the episode-level match-rate bookkeeping used by the + eco-evolution metrics.""" + agent = str(agent) + own_dialect = self.agent_live_dialect.get(agent) + is_predator = "predator" in agent + if own_dialect is None: + return 1.0 + majority = self._local_majority_dialect(agent) + if majority is None: + return 1.0 + matched = majority == own_dialect + if is_predator: + self.dialect_total_events_predator += 1 + if matched: + self.dialect_match_events_predator += 1 + else: + self.dialect_total_events_prey += 1 + if matched: + self.dialect_match_events_prey += 1 + return self.coordination_bonus_multiplier if matched else 1.0 + + def _build_live_culture_metrics(self) -> dict[str, float]: + """Genome/culture-state distributions of the currently alive population.""" + grouped: dict[str, list[str]] = {"predator": [], "prey": []} + for agent_id in self.agents: + agent_str = str(agent_id) + if agent_str in self.agent_genomes: + key = "predator" if "predator" in agent_str else "prey" + grouped[key].append(agent_str) + metrics: dict[str, float] = {} + for species, agent_ids in grouped.items(): + if agent_ids: + plasticity = [self.agent_genomes[a].plasticity for a in agent_ids] + live_dialects = [ + self.agent_live_dialect.get(a, self.agent_genomes[a].dialect) for a in agent_ids + ] + p25, p50, p75 = np.percentile(plasticity, [25, 50, 75]) + metrics[f"{species}_plasticity_mean"] = float(np.mean(plasticity)) + metrics[f"{species}_plasticity_std"] = float(np.std(plasticity)) + metrics[f"{species}_plasticity_p25"] = float(p25) + metrics[f"{species}_plasticity_p50"] = float(p50) + metrics[f"{species}_plasticity_p75"] = float(p75) + metrics[f"{species}_dialect_entropy"] = self._dialect_entropy(live_dialects) + metrics[f"{species}_count"] = float(len(agent_ids)) + else: + metrics[f"{species}_plasticity_mean"] = 0.0 + metrics[f"{species}_plasticity_std"] = 0.0 + metrics[f"{species}_plasticity_p25"] = 0.0 + metrics[f"{species}_plasticity_p50"] = 0.0 + metrics[f"{species}_plasticity_p75"] = 0.0 + metrics[f"{species}_dialect_entropy"] = 0.0 + metrics[f"{species}_count"] = 0.0 + return metrics + + def _dialect_entropy(self, dialects: list[int]) -> float: + """Shannon entropy (nats) of a live dialect distribution -- 0 when one + dialect has fixed in the population, ln(n_dialects) at maximum diversity.""" + if not dialects: + return 0.0 + counts = np.bincount(dialects, minlength=self.n_dialects).astype(np.float64) + total = counts.sum() + if total <= 0: + return 0.0 + probs = counts[counts > 0] / total + return float(-np.sum(probs * np.log(probs))) + + @staticmethod + def _spearman_corr(x: list, y: list) -> float: + """Dependency-free Spearman rank correlation with fractional + (average) tie ranking -- required since one caller's second input + (reproduced-or-not) is binary and ties heavily; ranking without tie + -averaging would make the result depend on arbitrary list order.""" + if len(x) < 2: + return 0.0 + x_arr = np.asarray(x, dtype=np.float64) + y_arr = np.asarray(y, dtype=np.float64) + + def _rank(a: np.ndarray) -> np.ndarray: + values, inverse, counts = np.unique(a, return_inverse=True, return_counts=True) + starts = np.concatenate(([0], np.cumsum(counts)[:-1])) + avg_rank_per_group = starts + (counts - 1) / 2.0 + return avg_rank_per_group[inverse] + + rx, ry = _rank(x_arr), _rank(y_arr) + if np.std(rx) == 0 or np.std(ry) == 0: + return 0.0 + return float(np.corrcoef(rx, ry)[0, 1]) + + def _policy_group(self, agent_id: str) -> str: + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unknown agent type in ID: {agent_id}") + + def _get_agent_genome(self, agent_id: str) -> Optional[Genome]: + return self.agent_genomes.get(agent_id) + + def _get_movement_energy_cost(self, agent_id: str, old_position, new_position) -> float: + distance = float(np.linalg.norm(np.array(new_position) - np.array(old_position))) + if distance <= 0: + return 0.0 + if "predator" in agent_id: + cost_per_cell = float(self.movement_energy_cost_per_cell_predator) + else: + cost_per_cell = float(self.movement_energy_cost_per_cell_prey) + return cost_per_cell * distance + + def _inherit_genome(self, agent_id: str, parent_agent_id: Optional[str], *, is_founder: bool) -> Optional[Genome]: + if not self.genome_enabled: + return None + if is_founder or parent_agent_id is None or parent_agent_id not in self.agent_genomes: + return founder_genome(self._policy_group(agent_id), self.genome_config, self.rng) + template_id = parent_agent_id + if self.genome_neutral_drift_control: + # Sever genome from reproductive success: the actual parent still pays + # the energy cost and determines spawn timing/position (population + # dynamics unchanged), but the mutation template is a uniformly random + # currently-alive same-species agent instead of the true parent. + species = self._policy_group(agent_id) + candidates = [ + str(aid) for aid in self.agents + if str(aid) in self.agent_genomes and self._policy_group(str(aid)) == species + ] + if candidates: + template_id = candidates[int(self.rng.integers(len(candidates)))] + return mutate_genome(self.agent_genomes[template_id], self.genome_config, self.rng) + + def _get_offspring_investment_energy(self, parent_agent_id: str) -> float: + return float(self.agent_energies[parent_agent_id]) * self.offspring_investment_fraction + + def _apply_time_step_update(self): + """ + Apply all per-step updates (energy decay, age increment). + """ + for raw_agent in list(self.agents): + agent = str(raw_agent) + is_predator = "predator" in agent + layer = 0 if is_predator else 1 + + # Basal metabolism always applies while the agent exists. + # Locomotion cost is charged later from actual distance moved. + if is_predator: + energy_decay = self.energy_loss_per_step_predator + else: + energy_decay = self.energy_loss_per_step_prey + + self.agent_energies[agent] -= energy_decay + self.grid_world_state[layer, *self.agent_positions[agent]] = self.agent_energies[agent] + + self.agent_ages[agent] += 1 + + self._per_agent_step_deltas[agent] = { + "decay": -energy_decay, + "move": 0.0, + "eat": 0.0, + "repro": 0.0, + } + + def _regenerate_grass_energy(self): + """ + Increase energy of all grass patches, capped at max_energy_grass. + """ + for grass, pos in self.grass_positions.items(): + old_energy = self.grass_energies[grass] + new_energy = min(old_energy + self.energy_gain_per_step_grass, self.max_energy_grass) + self.grass_energies[grass] = new_energy + self.grid_world_state[2, pos[0], pos[1]] = new_energy + + def _process_agent_movements(self, action_dict): + """ + Process movement and grid updates for all agents (non-vectorized, simple loop). + """ + for agent in action_dict.keys(): + if agent not in self.agent_positions or self.terminations.get(agent): + continue + old_position = self.agent_positions[agent] + action = action_dict[agent] + new_position = self._get_move(agent, action) + movement_cost = self._get_movement_energy_cost(agent, old_position, new_position) + self.agent_energies[agent] -= movement_cost + self._per_agent_step_deltas.setdefault( + agent, + {"decay": 0.0, "move": 0.0, "eat": 0.0, "repro": 0.0}, + ) + self._per_agent_step_deltas[agent]["move"] -= movement_cost + if "predator" in agent: + self.predator_positions[agent] = tuple(new_position) + self.grid_world_state[0, old_position[0], old_position[1]] = 0 + self.grid_world_state[0, new_position[0], new_position[1]] = self.agent_energies[agent] + elif "prey" in agent: + self.prey_positions[agent] = tuple(new_position) + self.grid_world_state[1, old_position[0], old_position[1]] = 0 + self.grid_world_state[1, new_position[0], new_position[1]] = self.agent_energies[agent] + record = self.agent_stats_live.get(agent) + if record is not None: + record["avg_energy_sum"] += self.agent_energies[agent] + record["avg_energy_steps"] += 1 + record["distance_traveled"] += float(np.linalg.norm(np.array(new_position) - np.array(old_position))) + record["movement_energy_spent"] += movement_cost + self.agent_positions[agent] = tuple(new_position) + + def _get_move(self, agent, action: int): + """ + Get the new position of the agent based on the action and its type. + """ + action = int(action) + + move_vector = self.action_to_move_tuple_agents[action] + + current_position = self.agent_positions[agent] + new_position = ( + current_position[0] + move_vector[0], + current_position[1] + move_vector[1], + ) + + # Clip new position to stay within grid bounds + new_position = tuple(np.clip(new_position, 0, self.grid_size - 1)) + + agent_layer = 0 if "predator" in agent else 1 + # Default: no block + self._last_move_block_reason[agent] = None + if self.grid_world_state[agent_layer, *new_position] > 0: + new_position = current_position + self._last_move_block_reason[agent] = "occupied" + + return new_position + + def _n_obs_channels(self) -> int: + return self.num_obs_channels + (2 if self.include_culture_in_obs else 0) + + def _get_observation(self, agent): + obs_range = self.predator_obs_range if "predator" in agent else self.prey_obs_range + xp, yp = self.agent_positions[agent] + xlo, xhi, ylo, yhi, xolo, xohi, yolo, yohi = self._obs_clip(xp, yp, obs_range) + n_ch = self._n_obs_channels() + obs = np.zeros((n_ch, obs_range, obs_range), dtype=np.float32) + obs[:self.num_obs_channels, xolo:xohi, yolo:yohi] = self.grid_world_state[:, xlo:xhi, ylo:yhi] + if self.include_culture_in_obs: + agent_str = str(agent) + own_dialect = self.agent_live_dialect.get(agent_str) + if own_dialect is not None: + dialect_norm = float(own_dialect) / max(float(self.n_dialects - 1), 1.0) + majority = self._local_majority_dialect(agent_str) + match_flag = 1.0 if (majority is not None and majority == own_dialect) else 0.0 + obs[self.num_obs_channels, :, :] = dialect_norm + obs[self.num_obs_channels + 1, :, :] = match_flag + return obs + + def _empty_observation(self, agent): + obs_range = self.predator_obs_range if "predator" in agent else self.prey_obs_range + n_ch = self._n_obs_channels() + return np.zeros((n_ch, obs_range, obs_range), dtype=np.float32) + + def _obs_clip(self, x, y, observation_range): + """ + Clip the observation window to the boundaries of the grid_world_state. + """ + observation_offset = (observation_range - 1) // 2 + xld, xhd = x - observation_offset, x + observation_offset + yld, yhd = y - observation_offset, y + observation_offset + xlo, xhi = np.clip(xld, 0, self.grid_size - 1), np.clip(xhd, 0, self.grid_size - 1) + ylo, yhi = np.clip(yld, 0, self.grid_size - 1), np.clip(yhd, 0, self.grid_size - 1) + xolo, yolo = abs(np.clip(xld, -observation_offset, 0)), abs(np.clip(yld, -observation_offset, 0)) + xohi, yohi = xolo + (xhi - xlo), yolo + (yhi - ylo) + return xlo, xhi + 1, ylo, yhi + 1, xolo, xohi + 1, yolo, yohi + 1 + + def _find_available_spawn_position(self, reference_position, occupied_positions): + """ + Finds an available position for spawning a new agent. + Tries to spawn near the parent agent first before selecting a random free position. + """ + x, y = reference_position + potential_positions = [ + (x + dx, y + dy) + for dx, dy in [(-1, 0), (1, 0), (0, -1), (0, 1)] + if 0 <= x + dx < self.grid_size and 0 <= y + dy < self.grid_size + ] + + # Filter for unoccupied positions + valid_positions = [pos for pos in potential_positions if pos not in occupied_positions] + + if valid_positions: + return valid_positions[0] # Prefer adjacent position if available + + # Fallback: Find any random unoccupied position + all_positions = { + (i, j) + for i in range(self.grid_size) + for j in range(self.grid_size) + } + free_positions = list(all_positions - occupied_positions) + + if free_positions: + return free_positions[self.rng.integers(len(free_positions))] + + return None + + def _handle_energy_starvation(self, agent): + self.observations[agent] = self._get_observation(agent) + self.rewards[agent] = 0 + self.terminations[agent] = True + self.truncations[agent] = False + + layer = 0 if "predator" in agent else 1 + self.grid_world_state[layer, *self.agent_positions[agent]] = 0 + self._finalize_agent_record(agent, cause="starved") + + if "predator" in agent: + self.active_num_predators -= 1 + else: + self.active_num_prey -= 1 + + def _handle_predator_engagement(self, agent): + self._per_agent_step_deltas.setdefault( + agent, + { + "decay": 0.0, + "move": 0.0, + "eat": 0.0, + "repro": 0.0, + }, + ) + predator_position = tuple(self.agent_positions[agent]) + caught_prey = next( + (prey for prey, pos in self.agent_positions.items() if "prey" in prey and np.array_equal(predator_position, pos)), None + ) + if caught_prey and self.current_step < self.agent_satiation_until.get(agent, 0): + # Still digesting/resting from a recent catch: does not hunt this + # step, so the prey survives. Regulates predator population growth + # through each predator's own recent hunting success rather than a + # population-level rule. + self.satiation_blocked_catches_predator += 1 + caught_prey = None + if caught_prey: + self.agents_just_ate.add(agent) + self.rewards[agent] = self._get_role_specific("reward_predator_catch_prey", agent) + prey_energy = float(self.agent_energies[caught_prey]) + # Satiation ceiling: a single kill can't provide more than this, + # regardless of how much energy the prey itself had accumulated. + energy_gain = min(prey_energy, self.max_energy_gain_per_prey) + energy_gain *= self._dialect_match_bonus(agent) + + self.agent_energies[agent] += energy_gain + self.grid_world_state[0, *predator_position] = self.agent_energies[agent] + self._per_agent_step_deltas[agent]["eat"] = energy_gain + if self.predator_satiation_cooldown > 0: + self.agent_satiation_until[agent] = self.current_step + self.predator_satiation_cooldown + predator_record = self.agent_stats_live.get(agent) + if predator_record is not None: + predator_record["times_ate"] += 1 + predator_record["energy_gained"] += energy_gain + predator_record["cumulative_reward"] += self.rewards[agent] + + # Prey is immediately terminated + self.observations[caught_prey] = self._get_observation(caught_prey) + self.terminations[caught_prey] = True + penalty = self._get_role_specific("penalty_prey_caught", caught_prey) + self.rewards[caught_prey] = penalty + self.truncations[caught_prey] = False + self.active_num_prey -= 1 + self.grid_world_state[1, *self.agent_positions[caught_prey]] = 0 + prey_record = self.agent_stats_live.get(caught_prey) + if prey_record is not None: + prey_record["death_cause"] = "eaten" + prey_record["cumulative_reward"] += penalty + self._finalize_agent_record(caught_prey, cause="eaten") + + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("eating_events", []).append( + { + "t": int(self.current_step), + "id_eaten": caught_prey, + "energy_after": float(self.agent_energies[agent]), + } + ) + else: + self.rewards[agent] = self._get_role_specific("reward_predator_step", agent) + predator_record = self.agent_stats_live.get(agent) + if predator_record is not None: + predator_record["cumulative_reward"] += self.rewards[agent] + + def _handle_prey_engagement(self, agent): + if self.terminations.get(agent): + return + prey_position = tuple(self.agent_positions[agent]) + caught_grass = next( + (g for g, pos in self.grass_positions.items() if "grass" in g and np.array_equal(prey_position, pos)), None + ) + if caught_grass: + self.agents_just_ate.add(agent) + self.rewards[agent] = self._get_role_specific("reward_prey_eat_grass", agent) + grass_energy = float(self.grass_energies[caught_grass]) + energy_gain = grass_energy + energy_gain *= self._dialect_match_bonus(agent) + + self.agent_energies[agent] += energy_gain + self._per_agent_step_deltas[agent]["eat"] = energy_gain + self.grid_world_state[1, *prey_position] = self.agent_energies[agent] + + self.grass_energies[caught_grass] = 0.0 + self.grid_world_state[2, *prey_position] = 0.0 + prey_record = self.agent_stats_live.get(agent) + if prey_record is not None: + prey_record["times_ate"] += 1 + prey_record["energy_gained"] += energy_gain + prey_record["cumulative_reward"] += self.rewards[agent] + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("eating_events", []).append( + { + "t": int(self.current_step), + "id_eaten": caught_grass, + "energy_after": float(self.agent_energies[agent]), + } + ) + else: + self.rewards[agent] = self._get_role_specific("reward_prey_step", agent) + prey_record = self.agent_stats_live.get(agent) + if prey_record is not None: + prey_record["cumulative_reward"] += self.rewards[agent] + + def _handle_predator_reproduction(self, agent): + self._per_agent_step_deltas.setdefault( + agent, + {"decay": 0.0, "move": 0.0, "eat": 0.0, "repro": 0.0}, + ) + if self.agent_energies[agent] < self.predator_creation_energy_threshold: + return + + new_agent = self._alloc_new_id("predator") + if not new_agent: + self.reproduction_blocked_due_to_capacity_predator += 1 + self.truncations["__all__"] = True + print( + f"[WARN] Predator ID pool exhausted at step {self.current_step} " + f"(n_possible_predators={self.n_possible_predators}). " + "Raise n_possible_predators in config." + ) + return + + self.agents.append(new_agent) + self._per_agent_step_deltas[new_agent] = { + "decay": 0.0, + "move": 0.0, + "eat": 0.0, + "repro": 0.0, + } + self.agent_last_reproduction[agent] = self.current_step + + self._register_new_agent(new_agent, parent_agent_id=str(agent)) + self.agent_live_offspring_ids[agent].append(new_agent) + self.agent_offspring_counts[agent] += 1 + parent_record = self.agent_stats_live.get(agent) + if parent_record is not None: + parent_record["offspring_count"] += 1 + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("reproduction_events", []).append( + {"t": int(self.current_step), "child_id": new_agent} + ) + + self.spawned_predators += 1 + + occupied_positions = set(self.agent_positions.values()) + new_position = self._find_available_spawn_position(self.agent_positions[agent], occupied_positions) + if new_position is None: + raise RuntimeError(f"No free spawn position available for predator offspring of {agent}.") + + self.agent_positions[new_agent] = new_position + self.predator_positions[new_agent] = new_position + + offspring_energy = self._get_offspring_investment_energy(agent) + self.agent_energies[new_agent] = offspring_energy + self.agent_energies[agent] -= offspring_energy + self._per_agent_step_deltas[agent]["repro"] = -offspring_energy + if parent_record is not None: + parent_record["reproduction_energy_invested_sum"] += offspring_energy + parent_record["reproduction_energy_invested_count"] += 1 + parent_record["parent_energy_after_reproduction_sum"] += self.agent_energies[agent] + parent_record["parent_energy_after_reproduction_count"] += 1 + child_record = self.agent_stats_live.get(new_agent) + if child_record is not None: + child_record["offspring_initial_energy"] = offspring_energy + + self.grid_world_state[0, *new_position] = offspring_energy + self.grid_world_state[0, *self.agent_positions[agent]] = self.agent_energies[agent] + + self.active_num_predators += 1 + + self.rewards[new_agent] = 0 + self.rewards[agent] = self._get_role_specific("reproduction_reward_predator", agent) + if parent_record is not None: + parent_record["cumulative_reward"] += self.rewards[agent] + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("reward_events", []).append( + { + "t": int(self.current_step), + "reproduction_reward": float(self.rewards[agent]), + "cumulative_reward": float(parent_record.get("cumulative_reward", 0.0)), + } + ) + + self.observations[new_agent] = self._get_observation(new_agent) + self.terminations[new_agent] = False + self.truncations[new_agent] = False + + def _handle_prey_reproduction(self, agent): + self._per_agent_step_deltas.setdefault( + agent, + {"decay": 0.0, "move": 0.0, "eat": 0.0, "repro": 0.0}, + ) + if self.agent_energies[agent] < self.prey_creation_energy_threshold: + return + + new_agent = self._alloc_new_id("prey") + if not new_agent: + self.reproduction_blocked_due_to_capacity_prey += 1 + self.truncations["__all__"] = True + print( + f"[WARN] Prey ID pool exhausted at step {self.current_step} " + f"(n_possible_prey={self.n_possible_prey}). " + "Raise n_possible_prey in config." + ) + return + + self.agents.append(new_agent) + self._per_agent_step_deltas[new_agent] = { + "decay": 0.0, + "move": 0.0, + "eat": 0.0, + "repro": 0.0, + } + + self.agent_last_reproduction[agent] = self.current_step + + self._register_new_agent(new_agent, parent_agent_id=str(agent)) + self.agent_live_offspring_ids[agent].append(new_agent) + + self.agent_offspring_counts[agent] += 1 + parent_record = self.agent_stats_live.get(agent) + if parent_record is not None: + parent_record["offspring_count"] += 1 + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("reproduction_events", []).append( + {"t": int(self.current_step), "child_id": new_agent} + ) + self.spawned_prey += 1 + + occupied_positions = set(self.agent_positions.values()) + new_position = self._find_available_spawn_position(self.agent_positions[agent], occupied_positions) + if new_position is None: + raise RuntimeError(f"No free spawn position available for prey offspring of {agent}.") + + self.agent_positions[new_agent] = new_position + self.prey_positions[new_agent] = new_position + + offspring_energy = self._get_offspring_investment_energy(agent) + self.agent_energies[new_agent] = offspring_energy + self.agent_energies[agent] -= offspring_energy + self._per_agent_step_deltas[agent]["repro"] = -offspring_energy + if parent_record is not None: + parent_record["reproduction_energy_invested_sum"] += offspring_energy + parent_record["reproduction_energy_invested_count"] += 1 + parent_record["parent_energy_after_reproduction_sum"] += self.agent_energies[agent] + parent_record["parent_energy_after_reproduction_count"] += 1 + child_record = self.agent_stats_live.get(new_agent) + if child_record is not None: + child_record["offspring_initial_energy"] = offspring_energy + + self.grid_world_state[1, *new_position] = offspring_energy + self.grid_world_state[1, *self.agent_positions[agent]] = self.agent_energies[agent] + + self.active_num_prey += 1 + + self.rewards[new_agent] = 0 + self.rewards[agent] = self._get_role_specific("reproduction_reward_prey", agent) + if parent_record is not None: + parent_record["cumulative_reward"] += self.rewards[agent] + evt = self.agent_event_log.get(agent) + if evt is not None: + evt.setdefault("reward_events", []).append( + { + "t": int(self.current_step), + "reproduction_reward": float(self.rewards[agent]), + "cumulative_reward": float(parent_record.get("cumulative_reward", 0.0)), + } + ) + + self.observations[new_agent] = self._get_observation(new_agent) + self.terminations[new_agent] = False + self.truncations[new_agent] = False + + def get_state_snapshot(self): + return { + "current_step": self.current_step, + "agent_positions": self.agent_positions.copy(), + "agent_energies": self.agent_energies.copy(), + "predator_positions": self.predator_positions.copy(), + "prey_positions": self.prey_positions.copy(), + "grass_positions": self.grass_positions.copy(), + "grass_energies": self.grass_energies.copy(), + "grid_world_state": self.grid_world_state.copy(), + "agents": self.agents.copy(), + "active_num_predators": self.active_num_predators, + "active_num_prey": self.active_num_prey, + "agents_just_ate": self.agents_just_ate.copy(), + "agent_stats_live": {aid: self._copy_agent_record(stats) for aid, stats in self.agent_stats_live.items()}, + "agent_stats_completed": { + aid: self._copy_agent_record(stats) for aid, stats in self.agent_stats_completed.items() + }, + "agent_ages": self.agent_ages.copy(), + "death_cause_prey": self.death_cause_prey.copy(), + "agent_last_reproduction": self.agent_last_reproduction.copy(), + "agent_parents": self.agent_parents.copy(), + "agent_genomes": { + aid: genome.to_dict() if genome is not None else None + for aid, genome in self.agent_genomes.items() + }, + "agent_live_dialect": self.agent_live_dialect.copy(), + "agent_offspring_counts": self.agent_offspring_counts.copy(), + "agent_live_offspring_ids": { + aid: list(ids) for aid, ids in self.agent_live_offspring_ids.items() + }, + "used_agent_ids": list(self.used_agent_ids), + "per_step_agent_data": self.per_step_agent_data.copy(), + } + + def restore_state_snapshot(self, snapshot): + self.current_step = snapshot["current_step"] + self.agent_positions = snapshot["agent_positions"].copy() + self.agent_energies = snapshot["agent_energies"].copy() + self.predator_positions = snapshot["predator_positions"].copy() + self.prey_positions = snapshot["prey_positions"].copy() + self.grass_positions = snapshot["grass_positions"].copy() + self.grass_energies = snapshot["grass_energies"].copy() + self.grid_world_state = snapshot["grid_world_state"].copy() + self.agents = snapshot["agents"].copy() + self.active_num_predators = snapshot["active_num_predators"] + self.active_num_prey = snapshot["active_num_prey"] + self.agents_just_ate = snapshot["agents_just_ate"].copy() + self.agent_stats_live = {aid: self._copy_agent_record(stats) for aid, stats in snapshot["agent_stats_live"].items()} + self.agent_stats_completed = { + aid: self._copy_agent_record(stats) for aid, stats in snapshot["agent_stats_completed"].items() + } + # Rebuild offspring lists in auxiliary index and ensure shared references + self.agent_live_offspring_ids = {} + for agent_id, record in self.agent_stats_live.items(): + offspring_list = list(record.get("offspring_ids", [])) + record["offspring_ids"] = offspring_list + self.agent_live_offspring_ids[agent_id] = offspring_list + self.agent_ages = snapshot["agent_ages"].copy() + self.death_cause_prey = snapshot["death_cause_prey"].copy() + self.agent_last_reproduction = snapshot["agent_last_reproduction"].copy() + self.agent_parents = snapshot.get("agent_parents", {}).copy() + active_genome_traits = set(Genome.__dataclass_fields__) + self.agent_genomes = { + aid: Genome(**{trait: genome_data.get(trait, GENOME_FIELD_DEFAULTS.get(trait)) for trait in active_genome_traits}) + for aid, genome_data in snapshot.get("agent_genomes", {}).items() + if genome_data is not None + } + self.agent_live_dialect = snapshot.get("agent_live_dialect", {}).copy() + self.agent_offspring_counts = snapshot.get("agent_offspring_counts", {}).copy() + stored_live_offspring = snapshot.get("agent_live_offspring_ids", {}) + if stored_live_offspring: + for agent_id, offspring_ids in stored_live_offspring.items(): + copied = list(offspring_ids) + if agent_id in self.agent_stats_live: + self.agent_stats_live[agent_id]["offspring_ids"] = copied + self.agent_live_offspring_ids[agent_id] = copied + self.used_agent_ids = set(snapshot.get("used_agent_ids", [])) + self.per_step_agent_data = snapshot["per_step_agent_data"].copy() + + def _build_possible_agent_ids(self): + agent_ids = [] + for i in range(self.n_possible_predators): + agent_ids.append(f"predator_{i}") + for i in range(self.n_possible_prey): + agent_ids.append(f"prey_{i}") + return agent_ids + + def _build_observation_space(self, agent_id): + """ + Build the observation space for a specific agent. + """ + n_ch = self._n_obs_channels() + if "predator" in agent_id: + obs_space = gymnasium.spaces.Box( + low=0, high=100.0, + shape=(n_ch, self.predator_obs_range, self.predator_obs_range), + dtype=np.float32, + ) + elif "prey" in agent_id: + obs_space = gymnasium.spaces.Box( + low=0, high=100.0, + shape=(n_ch, self.prey_obs_range, self.prey_obs_range), + dtype=np.float32, + ) + else: + raise ValueError(f"Unknown agent type in ID: {agent_id}") + return obs_space + + def _build_action_space(self, agent_id): + """ + Build the action space for a specific agent. + """ + if "predator" in agent_id or "prey" in agent_id: + action_space = gymnasium.spaces.Discrete(self.action_range**2) + else: + raise ValueError(f"Unknown agent type in ID: {agent_id}") + + return action_space + + def _register_new_agent(self, agent_id: str, parent_agent_id: Optional[str] = None, *, is_founder: bool = False): + if agent_id in self.agent_stats_live or agent_id in self.agent_stats_completed: + raise ValueError(f"Agent id {agent_id} already registered in this episode.") + self.used_agent_ids.add(agent_id) + self.agent_ages[agent_id] = 0 + self.agent_satiation_until[agent_id] = 0 + self.agent_offspring_counts[agent_id] = 0 + self.agent_live_offspring_ids[agent_id] = [] + self.agent_parents[agent_id] = parent_agent_id + genome = self._inherit_genome(agent_id, parent_agent_id, is_founder=is_founder) + if genome is not None: + self.agent_genomes[agent_id] = genome + genome_dict = genome.to_dict() + # Culture starts equal to the founder/inherited genetic bias, then + # can drift within this agent's own lifetime via + # _apply_cultural_learning, gated by the heritable plasticity trait. + self.agent_live_dialect[agent_id] = int(genome.dialect) + else: + genome_dict = None + # Initialize event-log entry + self.agent_event_log[agent_id] = { + "agent_id": agent_id, + "birth_step": self.current_step, + "death_step": None, + "parent_id": parent_agent_id, + "death_cause": None, + "eating_events": [], + "reproduction_events": [], + "reward_events": [], + "diet_events": [], + "lifecycle_events": [], + "genome": genome_dict, + } + self.agent_stats_live[agent_id] = { + "agent_id": agent_id, + "birth_step": self.current_step, + "parent": parent_agent_id, + "offspring_count": 0, + "offspring_ids": self.agent_live_offspring_ids[agent_id], + "distance_traveled": 0.0, + "movement_energy_spent": 0.0, + "times_ate": 0, + "energy_gained": 0.0, + "avg_energy_sum": 0.0, + "avg_energy_steps": 0, + "offspring_initial_energy": 0.0, + "reproduction_energy_invested_sum": 0.0, + "reproduction_energy_invested_count": 0, + "parent_energy_after_reproduction_sum": 0.0, + "parent_energy_after_reproduction_count": 0, + "cumulative_reward": 0.0, + "policy_group": self._policy_group(agent_id), + "genome": genome_dict, + "death_step": None, + "death_cause": None, + "avg_energy": 0.0, + } + + return self.agent_stats_live[agent_id] + + def _copy_agent_record(self, record: dict) -> dict: + copied = record.copy() + copied["offspring_ids"] = list(record.get("offspring_ids", [])) + return copied + + def _finalize_agent_record(self, agent_id: str, cause: Optional[str] = None): + record = self.agent_stats_live.pop(agent_id, None) + if record is None: + return + # Only add the final reward if not forcibly set (e.g., by engagement logic) + if not record.pop("_cumulative_reward_forced", False): + reward = self.rewards.get(agent_id, 0.0) + record["cumulative_reward"] += reward + if cause is not None: + record["death_cause"] = cause + + if record.get("death_step") is None: + record["death_step"] = self.current_step + record["offspring_count"] = self.agent_offspring_counts.get(agent_id, record.get("offspring_count", 0)) + steps = max(record.get("avg_energy_steps", 0), 1) + record["avg_energy"] = record.get("avg_energy_sum", 0.0) / steps + final_total = record.get("cumulative_reward", 0.0) + # Ensure callbacks can access the exact per-agent totals via infos regardless of termination path. + info = self._pending_infos.setdefault(agent_id, {}) + # Avoid overwriting if a branch (e.g., time-limit) already attached the same value. + info.setdefault("final_cumulative_reward", final_total) + birth_step = record.get("birth_step", self.current_step) + death_step = record.get("death_step", self.current_step) + lifetime = max(int(death_step - birth_step), 0) + info.setdefault("lifetime_steps", lifetime) + info.setdefault("parent_id", record.get("parent")) + # Mirror core lifecycle fields into the event log record for convenience + evt = self.agent_event_log.get(agent_id) + if evt is not None: + evt["death_step"] = death_step + evt["death_cause"] = record.get("death_cause") + evt.setdefault("parent_id", record.get("parent")) + if "predator" in agent_id: + info.setdefault("species", "predator") + elif "prey" in agent_id: + info.setdefault("species", "prey") + self.agent_stats_completed[agent_id] = record + self.agent_live_offspring_ids.pop(agent_id, None) + + def export_agent_event_log(self, path: str) -> None: + """Export the per-agent event log to a JSON file. + + Designed for evaluation scripts. Writes a dict keyed by agent_id, + where each value is that agent's event-log record. + """ + import json + + def _convert(obj): + if isinstance(obj, (int, float, str)) or obj is None: + return obj + if isinstance(obj, dict): + return {k: _convert(v) for k, v in obj.items()} + if isinstance(obj, (list, tuple)): + return [_convert(v) for v in obj] + try: + return obj.item() + except Exception: + return str(obj) + + payload = {aid: _convert(rec) for aid, rec in self.agent_event_log.items()} + with open(path, "w", encoding="utf-8") as f: + json.dump(payload, f, indent=2) + + def _attach_final_cumulative_rewards_for_live_agents(self): + """Ensure all agents that remain alive at episode end expose their cumulative rewards via infos.""" + for agent_id, record in self.agent_stats_live.items(): + info = self._pending_infos.setdefault(agent_id, {}) + info.setdefault("final_cumulative_reward", record.get("cumulative_reward", 0.0)) + birth_step = record.get("birth_step", self.current_step) + lifetime = max(int(self.current_step - birth_step), 0) + info.setdefault("lifetime_steps", lifetime) + info.setdefault("parent_id", record.get("parent")) + if "predator" in agent_id: + info.setdefault("species", "predator") + elif "prey" in agent_id: + info.setdefault("species", "prey") + + def _build_episode_training_metrics(self) -> dict[str, float]: + """Build lightweight scalar genome/behavior summaries for RLlib callbacks.""" + metrics = {} + grouped_ids = {"predator": [], "prey": []} + for agent_id, _record in self._iter_all_agent_records(): + if agent_id.startswith("predator"): + grouped_ids["predator"].append(agent_id) + elif agent_id.startswith("prey"): + grouped_ids["prey"].append(agent_id) + + for species, agent_ids in grouped_ids.items(): + genome_ids = [aid for aid in agent_ids if aid in self.agent_genomes] + if genome_ids: + plasticity = [self.agent_genomes[aid].plasticity for aid in genome_ids] + founder_dialects = [self.agent_genomes[aid].dialect for aid in genome_ids] + metrics[f"{species}_plasticity_mean"] = float(np.mean(plasticity)) + metrics[f"{species}_plasticity_std"] = float(np.std(plasticity)) + metrics[f"{species}_founder_dialect_entropy"] = self._dialect_entropy(founder_dialects) + reproduced_flags = [] + for aid in genome_ids: + record = self.agent_stats_live.get(aid) or self.agent_stats_completed.get(aid) + reproduced_flags.append(1.0 if record and record.get("offspring_count", 0) > 0 else 0.0) + metrics[f"{species}_plasticity_repro_spearman"] = self._spearman_corr(plasticity, reproduced_flags) + else: + metrics[f"{species}_plasticity_mean"] = 0.0 + metrics[f"{species}_plasticity_std"] = 0.0 + metrics[f"{species}_founder_dialect_entropy"] = 0.0 + metrics[f"{species}_plasticity_repro_spearman"] = 0.0 + + records = [ + record for aid, record in self._iter_all_agent_records() if aid in agent_ids + ] + if records: + metrics[f"{species}_distance_traveled_mean"] = float( + np.mean([float(record.get("distance_traveled", 0.0)) for record in records]) + ) + metrics[f"{species}_movement_energy_spent_mean"] = float( + np.mean([float(record.get("movement_energy_spent", 0.0)) for record in records]) + ) + metrics[f"{species}_offspring_count_mean"] = float( + np.mean([float(record.get("offspring_count", 0.0)) for record in records]) + ) + metrics[f"{species}_agent_count"] = float(len(records)) + metrics[f"{species}_offspring_initial_energy_mean"] = float( + np.mean([float(record.get("offspring_initial_energy", 0.0)) for record in records]) + ) + invested_values = [ + float(record.get("reproduction_energy_invested_sum", 0.0)) + / max(float(record.get("reproduction_energy_invested_count", 0.0)), 1.0) + for record in records + if float(record.get("reproduction_energy_invested_count", 0.0)) > 0.0 + ] + after_repro_values = [ + float(record.get("parent_energy_after_reproduction_sum", 0.0)) + / max(float(record.get("parent_energy_after_reproduction_count", 0.0)), 1.0) + for record in records + if float(record.get("parent_energy_after_reproduction_count", 0.0)) > 0.0 + ] + metrics[f"{species}_reproduction_energy_invested_mean"] = ( + float(np.mean(invested_values)) if invested_values else 0.0 + ) + metrics[f"{species}_parent_energy_after_reproduction_mean"] = ( + float(np.mean(after_repro_values)) if after_repro_values else 0.0 + ) + else: + metrics[f"{species}_distance_traveled_mean"] = 0.0 + metrics[f"{species}_movement_energy_spent_mean"] = 0.0 + metrics[f"{species}_offspring_count_mean"] = 0.0 + metrics[f"{species}_agent_count"] = 0.0 + metrics[f"{species}_offspring_initial_energy_mean"] = 0.0 + metrics[f"{species}_reproduction_energy_invested_mean"] = 0.0 + metrics[f"{species}_parent_energy_after_reproduction_mean"] = 0.0 + + metrics["predator_reproduction_blocked"] = float(self.reproduction_blocked_due_to_capacity_predator) + metrics["prey_reproduction_blocked"] = float(self.reproduction_blocked_due_to_capacity_prey) + metrics["predator_satiation_blocked_catches"] = float(self.satiation_blocked_catches_predator) + metrics["predator_spawned_total"] = float(self.spawned_predators) + metrics["prey_spawned_total"] = float(self.spawned_prey) + metrics["peak_active_predators"] = float(self.peak_active_predators) + metrics["peak_active_prey"] = float(self.peak_active_prey) + metrics["prey_unique_ids_used"] = float(sum(1 for a in self.used_agent_ids if "prey" in a)) + metrics["predator_unique_ids_used"] = float(sum(1 for a in self.used_agent_ids if "predator" in a)) + metrics["predator_dialect_match_rate"] = ( + float(self.dialect_match_events_predator) / self.dialect_total_events_predator + if self.dialect_total_events_predator else 0.0 + ) + metrics["prey_dialect_match_rate"] = ( + float(self.dialect_match_events_prey) / self.dialect_total_events_prey + if self.dialect_total_events_prey else 0.0 + ) + return metrics + + def _attach_episode_training_metrics(self): + info = self._pending_infos.setdefault("__all__", {}) + info["training_metrics"] = self._build_episode_training_metrics() + + def _iter_all_agent_records(self): + for agent_id, record in self.agent_stats_live.items(): + yield agent_id, record + for agent_id, record in self.agent_stats_completed.items(): + yield agent_id, record + + def get_all_agent_stats(self) -> dict[str, dict]: + """Return copies of all agent records keyed by agent_id.""" + stats_by_id = {} + for agent_id, record in self._iter_all_agent_records(): + copied = self._copy_agent_record(record) + stats_by_id[agent_id] = copied + return stats_by_id + + def get_total_energy_by_type(self): + """ + Returns a dict with total energy by category: + - 'predator': total predator energy + - 'prey': total prey energy + - 'grass': total grass energy + """ + energy_totals = { + "predator": 0.0, + "prey": 0.0, + "grass": sum(self.grass_energies.values()), + } + + for agent_id, energy in self.agent_energies.items(): + if "predator" in agent_id: + energy_totals["predator"] += energy + elif "prey" in agent_id: + energy_totals["prey"] += energy + + return energy_totals + + def get_total_offspring_by_type(self): + """Returns a dict of total offspring counts by role.""" + counts = {"predator": 0, "prey": 0} + for _, stats in self._iter_all_agent_records(): + group = stats.get("policy_group") + if group in counts: + counts[group] += stats.get("offspring_count", 0) + return counts + + def get_total_energy_spent_by_type(self): + """Returns a dict of total energy spent by role.""" + energy_spent = {"predator": 0.0, "prey": 0.0} + for _, stats in self._iter_all_agent_records(): + group = stats.get("policy_group") + if group in energy_spent: + energy_spent[group] += stats.get("energy_spent", 0.0) + return energy_spent + + def _get_role_specific(self, key: str, agent_id: str): + raw_val = getattr(self, f"{key}_config", 0.0) + if isinstance(raw_val, dict): + for k in raw_val: + if agent_id.startswith(k): + return raw_val[k] + raise KeyError(f"Type-specific key '{agent_id}' not found under '{key}'") + return raw_val + + #-------- Reset placement methods grid world entities -------- + def _create_and_place_grid_world_entities(self): + """ + Place and create all entities (predators, prey, grass) into the open grid world state. + """ + predator_list, prey_list, predator_positions, prey_positions, grass_positions = self._sample_agent_and_grass_positions() + self._place_predators(predator_list, predator_positions) + self._place_prey(prey_list, prey_positions) + self._place_grass(grass_positions) + + # -------- Reset other entity placement methods -------- + def _sample_agent_and_grass_positions(self): + """ + Sample free positions for all entities and return lists for placement. + Returns: + predator_list, prey_list, predator_positions, prey_positions, grass_positions + """ + num_grid_cells = self.grid_size * self.grid_size + num_agents_and_grass = len(self.agents) + len(self.grass_agents) + + chosen_grid_indices = self.rng.choice(num_grid_cells, size=num_agents_and_grass, replace=False) + chosen_positions = [(i // self.grid_size, i % self.grid_size) for i in chosen_grid_indices] + + predator_list = [str(a) for a in self.agents if "predator" in str(a)] + prey_list = [str(a) for a in self.agents if "prey" in str(a)] + + predator_positions = chosen_positions[: len(predator_list)] + prey_positions = chosen_positions[len(predator_list) : len(predator_list) + len(prey_list)] + grass_positions = chosen_positions[len(predator_list) + len(prey_list) :] + + return predator_list, prey_list, predator_positions, prey_positions, grass_positions + + #-------- Placement method for predators -------- + def _place_predators(self, predator_list, predator_positions): + self.predator_positions = {} + for i, agent in enumerate(predator_list): + pos = predator_positions[i] + self.agent_positions[agent] = pos + self.predator_positions[agent] = pos + self.agent_energies[agent] = self.initial_energy_predator_at_reset + self.grid_world_state[0, *pos] = self.initial_energy_predator_at_reset + + #-------- Placement method for prey -------- + def _place_prey(self, prey_list, prey_positions): + self.prey_positions = {} + for i, agent in enumerate(prey_list): + pos = prey_positions[i] + self.agent_positions[agent] = pos + self.prey_positions[agent] = pos + self.agent_energies[agent] = self.initial_energy_prey_at_reset + self.grid_world_state[1, *pos] = self.initial_energy_prey_at_reset + + #-------- Placement method for grass -------- + def _place_grass(self, grass_positions): + self.grass_positions = {} + self.grass_energies = {} + for i, grass in enumerate(self.grass_agents): + pos = grass_positions[i] + self.grass_positions[grass] = pos + self.grass_energies[grass] = self.initial_energy_grass + self.grid_world_state[2, *pos] = self.initial_energy_grass diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/random_policy.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/random_policy.py new file mode 100644 index 000000000..0cce8140c --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/random_policy.py @@ -0,0 +1,92 @@ +""" +Random policy for the PredPreyGrass environment. +No backward stepping is implemented in this version, +because that is pointless for debugging and testing +with a random policy. +""" + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.pygame_grid_renderer_rllib import PyGameRenderer + +# external libraries +import pygame +import random +import numpy as np + + + +def env_creator(config): + return PredPreyGrass(config) + + +def random_policy_pi(agent_id, env): + return env.action_spaces[agent_id].sample() + + +if __name__ == "__main__": + cfg = dict(config_env) + seed = cfg.get("seed") + if seed is None: + seed = random.SystemRandom().randint(0, 2**32 - 1) + cfg["seed"] = seed + random.seed(seed) + np.random.seed(seed) + env = env_creator(cfg) + observations, _ = env.reset(seed=seed) + + # Seed each action space with a distinct but reproducible seed to avoid + # agents sampling identical action sequences in lockstep. + action_spaces = env.action_spaces + assert action_spaces is not None + for i, (agent_id, space) in enumerate(action_spaces.items()): + space.seed(seed + i * 9973) + + # Debug: print one observation shape to confirm the configured channel count. + if observations: + first_agent = next(iter(observations)) + print(f"Sample observation shape for {first_agent}: {observations[first_agent].shape}") + + grid_size = (env.grid_size, env.grid_size) + visualizer = PyGameRenderer( + grid_size, + enable_speed_slider=False, + enable_tooltips=False, + predator_obs_range=cfg.get("predator_obs_range"), + prey_obs_range=cfg.get("prey_obs_range"), + show_fov=True, + fov_alpha=40, + fov_agents=["predator_0", "prey_0"], + ) + clock = pygame.time.Clock() + + # Run loop until termination + terminated = False + truncated = False + + while not terminated and not truncated: + # --- Step forward using random actions --- + action_dict = {agent_id: random_policy_pi(agent_id, env) for agent_id in env.agents} + env.step(action_dict) + # print(f"Step {env.current_step}") + # print(f"{terminations}") + + # --- Update visualizer --- + visualizer.update( + grass_positions=env.grass_positions, + grass_energies=env.grass_energies, + step=env.current_step, + agents_just_ate=env.agents_just_ate, + per_step_agent_data=env.per_step_agent_data, + dead_prey=getattr(env, "dead_prey", None), + ) + + terminated = env.terminations["__all__"] + truncated = env.truncations["__all__"] + + # Frame rate control + clock.tick(visualizer.target_fps) + + print(f"Episode ended at step: {env.current_step}") + visualizer.close() + env.close() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/resume_training_cultural_plasticity.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/resume_training_cultural_plasticity.py new file mode 100644 index 000000000..e5a974a46 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/resume_training_cultural_plasticity.py @@ -0,0 +1,139 @@ +""" +Resume PPO training for eco_evolutionary_cultural_plasticity from the latest checkpoint. + +Automatically finds the most recent PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_* experiment +directory under ~/ray_results/ and calls Tuner.restore() to continue training. +""" + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.episode_return_callback import EpisodeReturn +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.networks import build_multi_module_spec + +import ray +from ray.rllib.algorithms.ppo import PPOConfig +from ray.tune.registry import register_env +from ray.tune import Tuner + +import os +from pathlib import Path +from typing import Any + + +def get_config_ppo(): + num_cpus = os.cpu_count() + if num_cpus == 32: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_gpu_eco_evolutionary import config_ppo + else: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_cpu_eco_evolutionary import config_ppo + return config_ppo + + +def env_creator(config): + return PredPreyGrass(config) + + +def policy_mapping_fn(agent_id, *args, **kwargs): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unrecognized agent_id format: {agent_id}") + + +def find_latest_experiment(ray_results_path: Path, prefix: str) -> Path: + candidates = sorted( + [d for d in ray_results_path.iterdir() if d.is_dir() and d.name.startswith(prefix)], + key=lambda d: d.stat().st_mtime, + ) + if not candidates: + raise FileNotFoundError(f"No experiment directories found with prefix '{prefix}' in {ray_results_path}") + latest = candidates[-1] + print(f"Resuming from: {latest}") + return latest + + +# Higher entropy after resume encourages exploration during the cold-start phase before the +# ecosystem stabilizes. Drop back to config_ppo["entropy_coeff"] (0.01) once training +# recovers (typically after ~10-20 iterations). +RESUME_ENTROPY_COEFF = 0.05 + + +if __name__ == "__main__": + ray.shutdown() + ray.init(log_to_driver=True, ignore_reinit_error=True) + + register_env("PredPreyGrass", env_creator) + + ray_results_path = Path("~/ray_results/").expanduser() + experiment_path = find_latest_experiment(ray_results_path, "PPO_ECO_EVOLUTION_CULTURAL_PLASTICITY_") + + config_ppo = get_config_ppo() + + sample_env = env_creator(config=config_env) + assert sample_env.observation_spaces is not None and sample_env.action_spaces is not None + obs_by_policy: dict[str, Any] = {} + act_by_policy: dict[str, Any] = {} + for agent_id, obs_space in sample_env.observation_spaces.items(): + pid = policy_mapping_fn(agent_id) + if pid not in obs_by_policy: + obs_by_policy[pid] = obs_space + act_by_policy[pid] = sample_env.action_spaces[agent_id] + del sample_env + + multi_module_spec = build_multi_module_spec(obs_by_policy, act_by_policy) + policies = { + pid: (None, obs_by_policy[pid], act_by_policy[pid], {}) + for pid in obs_by_policy + } + + ppo_config = ( + PPOConfig() + .environment(env="PredPreyGrass", env_config=config_env, disable_env_checking=True) + .framework("torch") + .multi_agent( + policies=policies, + policy_mapping_fn=policy_mapping_fn, + ) + .training( + train_batch_size_per_learner=config_ppo["train_batch_size_per_learner"], + minibatch_size=config_ppo["minibatch_size"], + num_epochs=config_ppo["num_epochs"], + gamma=config_ppo["gamma"], + lr=config_ppo["lr"], + lambda_=config_ppo["lambda_"], + entropy_coeff=RESUME_ENTROPY_COEFF, + vf_loss_coeff=config_ppo["vf_loss_coeff"], + clip_param=config_ppo["clip_param"], + kl_coeff=config_ppo["kl_coeff"], + kl_target=config_ppo["kl_target"], + ) + .rl_module(rl_module_spec=multi_module_spec) + .learners( + num_gpus_per_learner=config_ppo["num_gpus_per_learner"], + num_learners=config_ppo["num_learners"], + ) + .env_runners( + num_env_runners=config_ppo["num_env_runners"], + num_envs_per_env_runner=config_ppo["num_envs_per_env_runner"], + rollout_fragment_length=config_ppo["rollout_fragment_length"], + sample_timeout_s=config_ppo["sample_timeout_s"], + num_cpus_per_env_runner=config_ppo["num_cpus_per_env_runner"], + ) + .resources( + num_cpus_for_main_process=config_ppo["num_cpus_for_main_process"], + ) + .callbacks(EpisodeReturn) + ) + + tuner = Tuner.restore( + path=str(experiment_path), + trainable=ppo_config.algo_class, # type: ignore[arg-type] + resume_unfinished=True, + resume_errored=False, + restart_errored=False, + param_space=ppo_config.to_dict(), + ) + + result = tuner.fit() + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/__init__.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/__init__.py new file mode 100644 index 000000000..c0e113dbd --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/__init__.py @@ -0,0 +1 @@ +"""Tests for the eco_evolutionary scenario.""" diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/test_eco_evolutionary_validation.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/test_eco_evolutionary_validation.py new file mode 100644 index 000000000..924b5efd2 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tests/test_eco_evolutionary_validation.py @@ -0,0 +1,642 @@ +import copy + +import numpy as np +import pytest + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.episode_return_callback import EpisodeReturn +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.genome import ( + Genome, + founder_genome, + mutate_genome, +) + + +def _make_test_env(overrides=None): + """Return a tiny eco_evolutionary env for deterministic unit tests.""" + cfg = copy.deepcopy(config_env) + cfg.update( + { + "n_initial_active_predators": 1, + "n_initial_active_prey": 1, + "n_possible_predators": 8, + "n_possible_prey": 8, + "initial_num_grass": 4, + } + ) + if overrides: + cfg.update(overrides) + # Pin min == max so initial population is deterministic in tests. + cfg["n_initial_active_predators_min"] = cfg["n_initial_active_predators"] + cfg["n_initial_active_prey_min"] = cfg["n_initial_active_prey"] + return PredPreyGrass(cfg) + + +def _place_agent(env, agent, position): + """Move one agent in both bookkeeping maps and grid-state channels.""" + old_position = env.agent_positions[agent] + if agent.startswith("predator"): + env.grid_world_state[0, *old_position] = 0.0 + env.predator_positions[agent] = position + env.grid_world_state[0, *position] = env.agent_energies[agent] + else: + env.grid_world_state[1, *old_position] = 0.0 + env.prey_positions[agent] = position + env.grid_world_state[1, *position] = env.agent_energies[agent] + env.agent_positions[agent] = position + + +class _FakeMetricsLogger: + def __init__(self): + self.values = {} + + def log_value(self, name, value): + self.values[name] = value + + +class _FakeAgentEpisode: + def __init__(self, total): + self._total = total + + def get_return(self): + return self._total + + +class _FakeEpisode: + length = 3 + agent_episodes = { + "predator_0": _FakeAgentEpisode(3.0), + "prey_0": _FakeAgentEpisode(-1.0), + } + + def get_return(self): + return 2.0 + + def get_rewards(self): + return { + "predator_0": [1.0, 2.0], + "prey_0": [-1.0], + } + + def get_last_infos(self): + return { + "__all__": { + "training_metrics": { + "predator_plasticity_mean": 0.15, + "predator_plasticity_repro_spearman": 0.2, + "prey_plasticity_mean": 0.08, + "prey_dialect_match_rate": 0.4, + "predator_movement_energy_spent_mean": 0.4, + "prey_offspring_count_mean": 2.0, + } + }, + "predator_0": { + "lifetime_steps": 3, + "final_cumulative_reward": 3.0, + }, + } + + +class _FakeEpisodeWithoutInfos(_FakeEpisode): + def get_last_infos(self): + return {} + + +class _FakeMetricsEnv: + def _build_episode_training_metrics(self): + return { + "predator_plasticity_mean": 0.12, + "prey_dialect_match_rate": 0.33, + } + + +class _FakeVectorEnv: + def __init__(self, envs): + self.envs = envs + + +# ---- Callback tests ---- + + +def test_episode_return_callback_logs_eco_evolution_metrics(): + callback = EpisodeReturn() + logger = _FakeMetricsLogger() + + callback.on_episode_end(episode=_FakeEpisode(), metrics_logger=logger) + + assert logger.values["eco_evolution/predator_plasticity_mean"] == pytest.approx(0.15) + assert logger.values["eco_evolution/predator_plasticity_repro_spearman"] == pytest.approx(0.2) + assert logger.values["eco_evolution/prey_plasticity_mean"] == pytest.approx(0.08) + assert logger.values["eco_evolution/prey_dialect_match_rate"] == pytest.approx(0.4) + assert logger.values["predator_episode_return_p50"] == pytest.approx(3.0) + + +def test_episode_return_callback_logs_eco_metrics_from_env_fallback(): + callback = EpisodeReturn() + logger = _FakeMetricsLogger() + + callback.on_episode_end( + episode=_FakeEpisodeWithoutInfos(), + metrics_logger=logger, + env=_FakeMetricsEnv(), + ) + + assert logger.values["eco_evolution/predator_plasticity_mean"] == pytest.approx(0.12) + assert logger.values["eco_evolution/prey_dialect_match_rate"] == pytest.approx(0.33) + + +def test_episode_return_callback_logs_eco_metrics_from_vector_env_fallback(): + callback = EpisodeReturn() + logger = _FakeMetricsLogger() + + callback.on_episode_end( + episode=_FakeEpisodeWithoutInfos(), + metrics_logger=logger, + env=_FakeVectorEnv([_FakeMetricsEnv()]), + env_index=0, + ) + + assert logger.values["eco_evolution/predator_plasticity_mean"] == pytest.approx(0.12) + assert logger.values["eco_evolution/prey_dialect_match_rate"] == pytest.approx(0.33) + + +# ---- RLlib contract tests (generic, not genome-specific) ---- + + +def test_every_acted_agent_gets_next_or_final_observation(): + env = _make_test_env( + { + "seed": 456, + "max_steps": 120, + "grid_size": 12, + "n_initial_active_predators": 4, + "n_initial_active_prey": 6, + "n_possible_predators": 80, + "n_possible_prey": 160, + "initial_num_grass": 30, + } + ) + observations, _ = env.reset(seed=456) + + for _ in range(10): + actions = { + agent: int(env.rng.integers(env.action_spaces[agent].n)) + for agent in observations + } + acted_agents = set(actions) + observations, _, terminations, truncations, _ = env.step(actions) + + for agent in acted_agents: + is_done = terminations.get(agent, False) or truncations.get(agent, False) + assert agent in observations or is_done + if is_done: + assert agent in observations + + if terminations.get("__all__") or truncations.get("__all__"): + break + + +def test_rllib_output_preserves_terminal_reward_without_terminal_observation(): + env = _make_test_env( + overrides={ + "predator_creation_energy_threshold": 999.0, + "prey_creation_energy_threshold": 999.0, + } + ) + env.reset(seed=124) + + predator = next(agent for agent in env.agents if agent.startswith("predator")) + prey = next(agent for agent in env.agents if agent.startswith("prey")) + _place_agent(env, predator, (5, 5)) + _place_agent(env, prey, (5, 5)) + + stay_action = next(i for i, move in env.action_to_move_tuple_agents.items() if move == (0, 0)) + observations, rewards, terminations, truncations, infos = env.step( + {agent: stay_action for agent in env.agents} + ) + + assert prey in observations + assert predator in observations + assert rewards[prey] == pytest.approx(env._get_role_specific("penalty_prey_caught", prey)) + assert terminations[prey] is True + assert truncations[prey] is False + assert terminations[predator] is True + assert truncations[predator] is False + assert terminations["__all__"] is True + assert truncations["__all__"] is False + assert "final_cumulative_reward" in infos[predator] + assert env.agents == [] + + +def test_time_limit_truncates_with_final_bootstrap_observations(): + env = _make_test_env( + overrides={ + "max_steps": 1, + "predator_creation_energy_threshold": 999.0, + "prey_creation_energy_threshold": 999.0, + } + ) + env.reset(seed=125) + + predator = next(agent for agent in env.agents if agent.startswith("predator")) + prey = next(agent for agent in env.agents if agent.startswith("prey")) + _place_agent(env, predator, (1, 1)) + _place_agent(env, prey, (env.grid_size - 2, env.grid_size - 2)) + + stay_action = next(i for i, move in env.action_to_move_tuple_agents.items() if move == (0, 0)) + observations, _, terminations, truncations, infos = env.step({agent: stay_action for agent in env.agents}) + + n_ch = env._n_obs_channels() + assert set(observations) == {predator, prey} + assert observations[predator].shape == (n_ch, env.predator_obs_range, env.predator_obs_range) + assert observations[prey].shape == (n_ch, env.prey_obs_range, env.prey_obs_range) + assert terminations[predator] is False + assert terminations[prey] is False + assert truncations[predator] is True + assert truncations[prey] is True + assert terminations["__all__"] is False + assert truncations["__all__"] is True + assert env.agents == [] + assert "training_metrics" in infos["__all__"] + metrics = infos["__all__"]["training_metrics"] + assert "predator_plasticity_mean" in metrics + assert "prey_plasticity_mean" in metrics + assert "predator_dialect_match_rate" in metrics + assert "prey_dialect_match_rate" in metrics + + +def test_action_space_uses_extended_moore_neighborhood(): + env = _make_test_env() + env.reset(seed=727) + assert env.action_spaces["predator_0"].n == 9 + assert (1, 0) in env.action_to_move_tuple_agents.values() + assert (2, 0) not in env.action_to_move_tuple_agents.values() + + +def test_observation_edges_are_clipped_and_zero_padded(): + env = _make_test_env() + env.reset(seed=808) + + predator = next(agent for agent in env.agents if agent.startswith("predator")) + env.agent_positions[predator] = (0, 0) + env.predator_positions[predator] = (0, 0) + + obs = env._get_observation(predator) + + assert obs.shape == (env._n_obs_channels(), env.predator_obs_range, env.predator_obs_range) + offset = (env.predator_obs_range - 1) // 2 + assert np.all(obs[:env.num_obs_channels, :offset, :] == 0.0) + assert np.all(obs[:env.num_obs_channels, :, :offset] == 0.0) + + +def test_insufficient_energy_blocks_reproduction_regardless_of_genome(): + env = _make_test_env(overrides={"predator_creation_energy_threshold": 10.0}) + env.reset(seed=88) + env.rewards = {} + + parent = next(a for a in env.agents if a.startswith("predator")) + env.agent_genomes[parent] = Genome(plasticity=0.5, dialect=0) + env.agent_energies[parent] = 9.99 # just below threshold + + env._handle_predator_reproduction(parent) + + assert env.agent_live_offspring_ids[parent] == [] + assert env.agent_offspring_counts[parent] == 0 + + +def test_energy_exactly_at_threshold_triggers_one_offspring(): + threshold = 10.0 + env = _make_test_env(overrides={"predator_creation_energy_threshold": threshold}) + env.reset(seed=99) + env.rewards = {} + + parent = next(a for a in env.agents if a.startswith("predator")) + env.agent_energies[parent] = threshold # exactly at threshold + + env._handle_predator_reproduction(parent) + + assert len(env.agent_live_offspring_ids[parent]) == 1 + + +def test_multi_generation_ancestry_chain(): + env = _make_test_env(overrides={"predator_creation_energy_threshold": 10.0}) + env.reset(seed=77) + env.rewards = {} + + parent = next(a for a in env.agents if a.startswith("predator")) + + # generation 1: parent -> child + env.agent_energies[parent] = 20.0 + env._handle_predator_reproduction(parent) + child = env.agent_live_offspring_ids[parent][0] + + # generation 2: child -> grandchild + env.agent_energies[child] = 20.0 + env._handle_predator_reproduction(child) + grandchild = env.agent_live_offspring_ids[child][0] + + assert env.agent_parents[child] == parent + assert env.agent_parents[grandchild] == child + assert grandchild in env.agent_genomes + assert grandchild in env.agent_live_dialect + + +def test_offspring_investment_fraction_is_fixed_and_genome_independent(): + fraction = 0.4 + env = _make_test_env( + overrides={"predator_creation_energy_threshold": 10.0, "offspring_investment_fraction": fraction} + ) + env.reset(seed=646) + env.rewards = {} + + parent = next(a for a in env.agents if a.startswith("predator")) + parent_energy = 20.0 + env.agent_energies[parent] = parent_energy + env.agent_genomes[parent] = Genome(plasticity=0.9, dialect=1) + + env._handle_predator_reproduction(parent) + child = env.agent_live_offspring_ids[parent][0] + + assert env.agent_energies[child] == pytest.approx(parent_energy * fraction) + assert env.agent_energies[parent] == pytest.approx(parent_energy * (1 - fraction)) + + +# ---- Genome / dual-inheritance tests ---- + + +def test_founders_receive_genomes_in_event_logs(): + env = _make_test_env() + env.reset(seed=515) + + predator = next(agent for agent in env.agents if agent.startswith("predator")) + + assert predator in env.agent_genomes + assert env.agent_stats_live[predator]["genome"] == env.agent_genomes[predator].to_dict() + assert env.agent_event_log[predator]["genome"] == env.agent_genomes[predator].to_dict() + assert predator in env.agent_live_dialect + assert env.agent_live_dialect[predator] == env.agent_genomes[predator].dialect + + +def test_genome_disabled_produces_no_genomes_and_no_live_dialect(): + env = _make_test_env(overrides={"genome_enabled": False}) + env.reset(seed=11) + + assert env.agent_genomes == {} + assert env.agent_live_dialect == {} + for agent in env.agents: + assert env.agent_event_log[agent]["genome"] is None + assert env.agent_stats_live[agent]["genome"] is None + + +def test_founder_genome_plasticity_respects_bounds(): + rng = np.random.default_rng(1) + config = { + "trait_bounds": {"plasticity": (0.0, 1.0)}, + "founder_genome": {"predator": {"plasticity_mean": 0.9, "plasticity_std": 0.5}}, + "n_dialects": 4, + } + for _ in range(200): + genome = founder_genome("predator", config, rng) + assert 0.0 <= genome.plasticity <= 1.0 + assert 0 <= genome.dialect < 4 + + +def test_zero_mutation_rate_produces_exact_genome_copy(): + rng = np.random.default_rng(3) + parent = Genome(plasticity=0.3, dialect=2) + config = {"genome_mutation": {"rate": 0.0}, "dialect_mutation": {"rate": 0.0}} + child = mutate_genome(parent, config, rng) + assert child.plasticity == pytest.approx(parent.plasticity) + assert child.dialect == parent.dialect + + +def test_mutation_always_stays_in_valid_state_space(): + rng = np.random.default_rng(4) + config = { + "trait_bounds": {"plasticity": (0.0, 1.0)}, + "genome_mutation": {"rate": 1.0, "std": 0.5}, + "dialect_mutation": {"rate": 1.0}, + "n_dialects": 5, + } + genome = Genome(plasticity=0.5, dialect=0) + for _ in range(200): + genome = mutate_genome(genome, config, rng) + assert 0.0 <= genome.plasticity <= 1.0 + assert 0 <= genome.dialect < 5 + + +def test_neutral_drift_control_template_is_a_live_conspecific_not_necessarily_parent(): + env = _make_test_env( + overrides={ + "predator_creation_energy_threshold": 10.0, + "n_initial_active_predators": 2, + "genome_neutral_drift_control": True, + "genome_mutation": {"rate": 0.0}, + "dialect_mutation": {"rate": 0.0}, + } + ) + env.reset(seed=656) + env.rewards = {} + + predators = [a for a in env.agents if a.startswith("predator")] + assert len(predators) == 2 + env.agent_genomes[predators[0]] = Genome(plasticity=0.9, dialect=0) + env.agent_genomes[predators[1]] = Genome(plasticity=0.1, dialect=3) + + env.agent_energies[predators[0]] = 20.0 + env._handle_predator_reproduction(predators[0]) + child = env.agent_live_offspring_ids[predators[0]][0] + + # With zero mutation, the child's genome must be an exact copy of ONE of the + # two live conspecifics' genomes -- not necessarily the reproducing parent's, + # since the template is a uniformly random live conspecific under the control. + possible = { + (env.agent_genomes[predators[0]].plasticity, env.agent_genomes[predators[0]].dialect), + (env.agent_genomes[predators[1]].plasticity, env.agent_genomes[predators[1]].dialect), + } + assert (env.agent_genomes[child].plasticity, env.agent_genomes[child].dialect) in possible + + +def test_live_culture_metrics_reflect_actual_genomes(): + env = _make_test_env() + env.reset(seed=111) + + for agent in env.agents: + env.agent_genomes[agent] = Genome(plasticity=0.25, dialect=1) + env.agent_live_dialect[agent] = 1 + + metrics = env._build_live_culture_metrics() + + assert metrics["predator_plasticity_mean"] == pytest.approx(0.25) + assert metrics["prey_plasticity_mean"] == pytest.approx(0.25) + assert metrics["predator_count"] == pytest.approx(1.0) + + +def test_dialect_entropy_zero_when_fixed_and_positive_when_diverse(): + env = _make_test_env() + env.reset(seed=222) + + assert env._dialect_entropy([2, 2, 2, 2]) == pytest.approx(0.0) + assert env._dialect_entropy([]) == pytest.approx(0.0) + diverse_entropy = env._dialect_entropy([0, 1, 2, 3]) + assert diverse_entropy > 0.0 + # Uniform distribution over n_dialects maximizes entropy at ln(n_dialects). + assert diverse_entropy == pytest.approx(np.log(4), rel=1e-6) + + +def test_spearman_corr_perfect_and_no_variance_cases(): + env = _make_test_env() + env.reset(seed=333) + + assert env._spearman_corr([1, 2, 3, 4], [1, 2, 3, 4]) == pytest.approx(1.0) + assert env._spearman_corr([1, 2, 3, 4], [4, 3, 2, 1]) == pytest.approx(-1.0) + assert env._spearman_corr([1, 1, 1], [1, 2, 3]) == pytest.approx(0.0) + assert env._spearman_corr([1], [1]) == pytest.approx(0.0) + + +def test_local_majority_dialect_none_without_same_species_neighbors(): + env = _make_test_env(overrides={"culture_range": 3, "n_initial_active_predators": 1, "n_initial_active_prey": 1}) + env.reset(seed=444) + + predator = next(a for a in env.agents if a.startswith("predator")) + _place_agent(env, predator, (5, 5)) + assert env._local_majority_dialect(predator) is None + + +def test_local_majority_dialect_picks_most_common_neighbor_dialect(): + # _local_majority_dialect excludes the focal agent's own vote (the norm + # to conform to is defined by *others*), so 4 agents are used here: 3 + # sharing dialect 2 and 1 with dialect 0, giving an unambiguous 2-vs-1 + # neighbor majority for every agent regardless of which one is focal. + env = _make_test_env( + overrides={ + "culture_range": 5, + "n_initial_active_predators": 4, + "n_initial_active_prey": 1, + } + ) + env.reset(seed=555) + + predators = [a for a in env.agents if a.startswith("predator")] + for i, agent in enumerate(predators): + _place_agent(env, agent, (5, 5 + i)) + env.agent_live_dialect[predators[0]] = 2 + env.agent_live_dialect[predators[1]] = 2 + env.agent_live_dialect[predators[2]] = 0 + env.agent_live_dialect[predators[3]] = 2 + + assert env._local_majority_dialect(predators[2]) == 2 + assert env._local_majority_dialect(predators[0]) == 2 + + +def test_cultural_learning_respects_check_interval_and_plasticity(): + env = _make_test_env( + overrides={ + "culture_range": 5, + "plasticity_check_interval": 4, + "n_initial_active_predators": 2, + "n_initial_active_prey": 1, + } + ) + env.reset(seed=666) + + predators = [a for a in env.agents if a.startswith("predator")] + for i, agent in enumerate(predators): + _place_agent(env, agent, (5, 5 + i)) + env.agent_genomes[predators[0]] = Genome(plasticity=1.0, dialect=0) # always adopts + env.agent_live_dialect[predators[0]] = 0 + env.agent_live_dialect[predators[1]] = 3 + + env.current_step = 1 # not a multiple of the check interval (4): no-op + env._apply_cultural_learning() + assert env.agent_live_dialect[predators[0]] == 0 + + env.current_step = 4 # a check step: predators[0] should adopt predators[1]'s dialect + env._apply_cultural_learning() + assert env.agent_live_dialect[predators[0]] == 3 + + +def test_zero_plasticity_never_adopts_majority_dialect(): + env = _make_test_env( + overrides={ + "culture_range": 5, + "plasticity_check_interval": 1, + "n_initial_active_predators": 2, + "n_initial_active_prey": 1, + } + ) + env.reset(seed=777) + + predators = [a for a in env.agents if a.startswith("predator")] + for i, agent in enumerate(predators): + _place_agent(env, agent, (5, 5 + i)) + env.agent_genomes[predators[0]] = Genome(plasticity=0.0, dialect=0) # never adopts + env.agent_live_dialect[predators[0]] = 0 + env.agent_live_dialect[predators[1]] = 3 + + env.current_step = 1 + for _ in range(20): + env._apply_cultural_learning() + + assert env.agent_live_dialect[predators[0]] == 0 + + +def test_dialect_match_grants_coordination_bonus_on_grass_energy_gain(): + bonus = 2.0 + env = _make_test_env( + overrides={"coordination_bonus_multiplier": bonus, "culture_range": 5, "n_initial_active_prey": 2} + ) + env.reset(seed=888) + env.rewards = {} + + prey_agents = [a for a in env.agents if a.startswith("prey")] + assert len(prey_agents) == 2 + + prey = prey_agents[0] + neighbor = prey_agents[1] + _place_agent(env, prey, (3, 3)) + _place_agent(env, neighbor, (3, 4)) + env.agent_live_dialect[prey] = 1 + env.agent_live_dialect[neighbor] = 1 # matches -> majority is dialect 1 + + grass_id = next(iter(env.grass_positions)) + env.grass_positions[grass_id] = (3, 3) + env.grass_energies[grass_id] = 1.5 + env.grid_world_state[2, 3, 3] = 1.5 + + env._apply_time_step_update() + start_energy = env.agent_energies[prey] + env._handle_prey_engagement(prey) + + assert env.agent_energies[prey] == pytest.approx(start_energy + 1.5 * bonus) + + +def test_dialect_mismatch_grants_no_coordination_bonus(): + bonus = 2.0 + env = _make_test_env( + overrides={"coordination_bonus_multiplier": bonus, "culture_range": 5, "n_initial_active_prey": 2} + ) + env.reset(seed=999) + env.rewards = {} + + prey_agents = [a for a in env.agents if a.startswith("prey")] + assert len(prey_agents) == 2 + + prey = prey_agents[0] + neighbor = prey_agents[1] + _place_agent(env, prey, (3, 3)) + _place_agent(env, neighbor, (3, 4)) + env.agent_live_dialect[prey] = 0 + env.agent_live_dialect[neighbor] = 1 # neighbor's dialect is the (sole) majority -> mismatch for prey + + grass_id = next(iter(env.grass_positions)) + env.grass_positions[grass_id] = (3, 3) + env.grass_energies[grass_id] = 1.5 + env.grid_world_state[2, 3, 3] = 1.5 + + env._apply_time_step_update() + start_energy = env.agent_energies[prey] + env._handle_prey_engagement(prey) + + assert env.agent_energies[prey] == pytest.approx(start_energy + 1.5) diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity.py new file mode 100644 index 000000000..751a2c456 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity.py @@ -0,0 +1,198 @@ +""" +Trains the eco_evolutionary_cultural_plasticity environment with PPO using the Ray RLlib +new API stack. + +The environment is a predator-prey-grass grid world implementing gene-culture +coevolution (dual inheritance, Boyd & Richerson) rather than a single genetic +channel. Agents carry a heritable `plasticity` trait (the Darwinian gene) that +sets how readily their live `dialect` -- an arbitrary same-species coordination +code, culturally transmitted via imitation of local same-species neighbors -- +updates toward the local majority. Dialect-matching at a catch/graze event +grants a coordination-game energy bonus. The social-learning update is a +genuine per-individual lifetime process, independent of the shared PPO policy +that governs movement/hunting behavior -- the plasticity gene has leverage +over how fast that process runs, not over behavior directly. + +Checkpoints and a copy of the environment source are saved under ~/ray_results/ +for provenance. Genome-trait statistics are logged to TensorBoard via the +EpisodeReturn callback. +""" + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.episode_return_callback import EpisodeReturn +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.networks import build_multi_module_spec + +import ray +from ray.rllib.algorithms.ppo import PPOConfig +from ray.tune.registry import register_env +from ray.tune import Tuner, RunConfig, CheckpointConfig + +import argparse +import os +from datetime import datetime +from pathlib import Path +import json +import shutil +from typing import Any + + +def parse_args(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--seed", type=int, default=None, + help="Override config_env['seed'] for this run (for multi-seed replication). " + "Tags the experiment name so runs don't collide.", + ) + parser.add_argument( + "--max-iters", type=int, default=None, + help="Override config_ppo['max_iters'] for this run.", + ) + return parser.parse_args() + + +def get_config_ppo(): + num_cpus = os.cpu_count() + if num_cpus == 32: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_gpu_eco_evolutionary import config_ppo + elif num_cpus == 8: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_cpu_eco_evolutionary import config_ppo + else: + # Default to CPU config for other CPU counts to keep training usable across machines. + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_cpu_eco_evolutionary import config_ppo + return config_ppo + + +def env_creator(config): + return PredPreyGrass(config) + + +def policy_mapping_fn(agent_id, *args, **kwargs): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unrecognized agent_id format: {agent_id}") + + +# --- Main training setup --- + +if __name__ == "__main__": + args = parse_args() + if args.seed is not None: + config_env = dict(config_env, seed=args.seed) + + ray.shutdown() + ray.init(log_to_driver=True, ignore_reinit_error=True) + + register_env("PredPreyGrass", env_creator) + + ray_results_dir = "~/ray_results/" + ray_results_path = Path(ray_results_dir).expanduser() + timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + seed_tag = f"_SEED{args.seed}" if args.seed is not None else "" + version = f"ECO_EVOLUTION_CULTURAL_PLASTICITY{seed_tag}" + experiment_name = f"PPO_{version}_{timestamp}" + experiment_path = ray_results_path / experiment_name + + experiment_path.mkdir(parents=True, exist_ok=True) + # --- Save environment source file for provenance --- + source_dir = experiment_path / "SOURCE_CODE" + source_dir.mkdir(exist_ok=True) + env_file = Path(__file__).parent / "predpreygrass_rllib_env.py" + shutil.copy2(env_file, source_dir / f"predpreygrass_rllib_env_{version}.py") + + config_ppo = get_config_ppo() + if args.max_iters is not None: + config_ppo = dict(config_ppo, max_iters=args.max_iters) + config_metadata = { + "config_env": config_env, + "config_ppo": config_ppo, + } + with open(experiment_path / "run_config.json", "w") as f: + json.dump(config_metadata, f, indent=4) + + sample_env = env_creator(config=config_env) + if sample_env.observation_spaces is None or sample_env.action_spaces is None: + raise RuntimeError("PredPreyGrass must define observation_spaces and action_spaces for all policies.") + + # Group spaces per policy id (first agent of each policy defines the space) + obs_by_policy: dict[str, Any] = {} + act_by_policy: dict[str, Any] = {} + for agent_id, obs_space in sample_env.observation_spaces.items(): + pid = policy_mapping_fn(agent_id) + if pid not in obs_by_policy: + obs_by_policy[pid] = obs_space + act_by_policy[pid] = sample_env.action_spaces[agent_id] + + # Build one MultiRLModuleSpec in one go + multi_module_spec = build_multi_module_spec(obs_by_policy, act_by_policy) + + # Policies dict for RLlib + policies = { + pid: (None, obs_by_policy[pid], act_by_policy[pid], {}) + for pid in obs_by_policy + } + + # Build config dictionary for Tune + ppo_config = ( + PPOConfig() + .environment(env="PredPreyGrass", env_config=config_env, disable_env_checking=True) + .framework("torch") + .multi_agent( + policies=policies, + policy_mapping_fn=policy_mapping_fn, + ) + .training( + train_batch_size_per_learner=config_ppo["train_batch_size_per_learner"], + minibatch_size=config_ppo["minibatch_size"], + num_epochs=config_ppo["num_epochs"], + gamma=config_ppo["gamma"], + lr=config_ppo["lr"], + lambda_=config_ppo["lambda_"], + entropy_coeff=config_ppo["entropy_coeff"], + vf_loss_coeff=config_ppo["vf_loss_coeff"], + clip_param=config_ppo["clip_param"], + kl_coeff=config_ppo["kl_coeff"], + kl_target=config_ppo["kl_target"], + ) + .rl_module(rl_module_spec=multi_module_spec) + .learners( + num_gpus_per_learner=config_ppo["num_gpus_per_learner"], + num_learners=config_ppo["num_learners"], + ) + .env_runners( + num_env_runners=config_ppo["num_env_runners"], + num_envs_per_env_runner=config_ppo["num_envs_per_env_runner"], + rollout_fragment_length=config_ppo["rollout_fragment_length"], + sample_timeout_s=config_ppo["sample_timeout_s"], + num_cpus_per_env_runner=config_ppo["num_cpus_per_env_runner"], + ) + + .resources( + num_cpus_for_main_process=config_ppo["num_cpus_for_main_process"], + ) + .callbacks(EpisodeReturn) + ) + + max_iters = config_ppo["max_iters"] + checkpoint_every = 10 + del sample_env # to avoid any stray references + + tuner = Tuner( + ppo_config.algo_class, + param_space=ppo_config.to_dict(), + run_config=RunConfig( + name=experiment_name, + storage_path=str(ray_results_path), + stop={"training_iteration": max_iters}, + checkpoint_config=CheckpointConfig( + num_to_keep=100, + checkpoint_frequency=checkpoint_every, + checkpoint_at_end=True, + ), + ), + ) + + result = tuner.fit() + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity_neutral_control.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity_neutral_control.py new file mode 100644 index 000000000..4a4f6b71e --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/tune_ppo_cultural_plasticity_neutral_control.py @@ -0,0 +1,193 @@ +""" +Neutral-drift control for the eco_evolutionary_cultural_plasticity environment. + +Identical to tune_ppo_cultural_plasticity.py in every respect except the +environment config: genome_neutral_drift_control=True severs genome +inheritance from reproductive success (offspring genome templates come from a +random currently-alive same-species agent, not the actual parent) while +leaving population/energy dynamics unchanged. Any plasticity-trait drift +observed in this run is attributable purely to mutation + finite-population +sampling noise, not selection. + +Compare this run's live_culture/*_plasticity_mean trajectory directly against +the real run (see RESULTS.md). +""" + +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.predpreygrass_rllib_env import PredPreyGrass +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_env_eco_evolutionary_neutral_control import config_env +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.episode_return_callback import EpisodeReturn +from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.utils.networks import build_multi_module_spec + +import ray +from ray.rllib.algorithms.ppo import PPOConfig +from ray.tune.registry import register_env +from ray.tune import Tuner, RunConfig, CheckpointConfig + +import argparse +import os +from datetime import datetime +from pathlib import Path +import json +import shutil +from typing import Any + + +def parse_args(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--seed", type=int, default=None, + help="Override config_env['seed'] for this run (for multi-seed replication). " + "Tags the experiment name so runs don't collide.", + ) + parser.add_argument( + "--max-iters", type=int, default=None, + help="Override config_ppo['max_iters'] for this run.", + ) + return parser.parse_args() + + +def get_config_ppo(): + num_cpus = os.cpu_count() + if num_cpus == 32: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_gpu_eco_evolutionary import config_ppo + elif num_cpus == 8: + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_cpu_eco_evolutionary import config_ppo + else: + # Default to CPU config for other CPU counts to keep training usable across machines. + from predpreygrass.evolutionary.eco_evolutionary_cultural_plasticity.config.config_ppo_cpu_eco_evolutionary import config_ppo + return config_ppo + + +def env_creator(config): + return PredPreyGrass(config) + + +def policy_mapping_fn(agent_id, *args, **kwargs): + if "predator" in agent_id: + return "predator" + if "prey" in agent_id: + return "prey" + raise ValueError(f"Unrecognized agent_id format: {agent_id}") + + +# --- Main training setup --- + +if __name__ == "__main__": + args = parse_args() + if args.seed is not None: + config_env = dict(config_env, seed=args.seed) + + ray.shutdown() + ray.init(log_to_driver=True, ignore_reinit_error=True) + + register_env("PredPreyGrass", env_creator) + + ray_results_dir = "~/ray_results/" + ray_results_path = Path(ray_results_dir).expanduser() + timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + seed_tag = f"_SEED{args.seed}" if args.seed is not None else "" + version = f"ECO_EVOLUTION_CULTURAL_PLASTICITY_NEUTRAL_CONTROL{seed_tag}" + experiment_name = f"PPO_{version}_{timestamp}" + experiment_path = ray_results_path / experiment_name + + experiment_path.mkdir(parents=True, exist_ok=True) + # --- Save environment source file for provenance --- + source_dir = experiment_path / "SOURCE_CODE" + source_dir.mkdir(exist_ok=True) + env_file = Path(__file__).parent / "predpreygrass_rllib_env.py" + shutil.copy2(env_file, source_dir / f"predpreygrass_rllib_env_{version}.py") + + config_ppo = get_config_ppo() + if args.max_iters is not None: + config_ppo = dict(config_ppo, max_iters=args.max_iters) + config_metadata = { + "config_env": config_env, + "config_ppo": config_ppo, + } + with open(experiment_path / "run_config.json", "w") as f: + json.dump(config_metadata, f, indent=4) + + sample_env = env_creator(config=config_env) + if sample_env.observation_spaces is None or sample_env.action_spaces is None: + raise RuntimeError("PredPreyGrass must define observation_spaces and action_spaces for all policies.") + + # Group spaces per policy id (first agent of each policy defines the space) + obs_by_policy: dict[str, Any] = {} + act_by_policy: dict[str, Any] = {} + for agent_id, obs_space in sample_env.observation_spaces.items(): + pid = policy_mapping_fn(agent_id) + if pid not in obs_by_policy: + obs_by_policy[pid] = obs_space + act_by_policy[pid] = sample_env.action_spaces[agent_id] + + # Build one MultiRLModuleSpec in one go + multi_module_spec = build_multi_module_spec(obs_by_policy, act_by_policy) + + # Policies dict for RLlib + policies = { + pid: (None, obs_by_policy[pid], act_by_policy[pid], {}) + for pid in obs_by_policy + } + + # Build config dictionary for Tune + ppo_config = ( + PPOConfig() + .environment(env="PredPreyGrass", env_config=config_env, disable_env_checking=True) + .framework("torch") + .multi_agent( + policies=policies, + policy_mapping_fn=policy_mapping_fn, + ) + .training( + train_batch_size_per_learner=config_ppo["train_batch_size_per_learner"], + minibatch_size=config_ppo["minibatch_size"], + num_epochs=config_ppo["num_epochs"], + gamma=config_ppo["gamma"], + lr=config_ppo["lr"], + lambda_=config_ppo["lambda_"], + entropy_coeff=config_ppo["entropy_coeff"], + vf_loss_coeff=config_ppo["vf_loss_coeff"], + clip_param=config_ppo["clip_param"], + kl_coeff=config_ppo["kl_coeff"], + kl_target=config_ppo["kl_target"], + ) + .rl_module(rl_module_spec=multi_module_spec) + .learners( + num_gpus_per_learner=config_ppo["num_gpus_per_learner"], + num_learners=config_ppo["num_learners"], + ) + .env_runners( + num_env_runners=config_ppo["num_env_runners"], + num_envs_per_env_runner=config_ppo["num_envs_per_env_runner"], + rollout_fragment_length=config_ppo["rollout_fragment_length"], + sample_timeout_s=config_ppo["sample_timeout_s"], + num_cpus_per_env_runner=config_ppo["num_cpus_per_env_runner"], + ) + + .resources( + num_cpus_for_main_process=config_ppo["num_cpus_for_main_process"], + ) + .callbacks(EpisodeReturn) + ) + + max_iters = config_ppo["max_iters"] + checkpoint_every = 10 + del sample_env # to avoid any stray references + + tuner = Tuner( + ppo_config.algo_class, + param_space=ppo_config.to_dict(), + run_config=RunConfig( + name=experiment_name, + storage_path=str(ray_results_path), + stop={"training_iteration": max_iters}, + checkpoint_config=CheckpointConfig( + num_to_keep=100, + checkpoint_frequency=checkpoint_every, + checkpoint_at_end=True, + ), + ), + ) + + result = tuner.fit() + ray.shutdown() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/episode_return_callback.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/episode_return_callback.py new file mode 100644 index 000000000..5d65e0876 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/episode_return_callback.py @@ -0,0 +1,282 @@ +from ray.rllib.callbacks.callbacks import RLlibCallback +from ray.rllib.utils.metrics.metrics_logger import MetricsLogger +import time +from collections import defaultdict +from typing import Any, Optional +import numpy as np + + +class EpisodeReturn(RLlibCallback): + def __init__(self): + super().__init__() + self.overall_sum_of_rewards = 0.0 + self.num_episodes = 0 + self._pending_episode_metrics = [] + self.start_time = time.time() + self.last_iteration_time = self.start_time + # rely on built-in RLlib episode length metrics instead of manual counting + + def on_episode_end( + self, + *, + episode, + metrics_logger: Optional[MetricsLogger] = None, + env=None, + env_index: int = 0, + **kwargs, + ): + """ + Called at the end of each episode. + Logs the total and average rewards separately for predators and prey. + """ + self.num_episodes += 1 + episode_return = episode.get_return() + episode_length = episode.env_steps() if callable(getattr(episode, "env_steps", None)) else 0 + self.overall_sum_of_rewards += episode_return + + def log_value(name: str, value: float): + if metrics_logger is not None: + metrics_logger.log_value(name, float(value)) + + # Accumulate rewards by group + group_rewards = defaultdict(list) + predator_total = prey_total = 0.0 + predator_totals = [] + prey_totals = [] + + # episode.get_rewards() indexes a global-step-aligned lookback buffer that + # can go out of range for agents whose local step count diverges from the + # episode's (e.g. short-lived offspring under a low fixed investment + # fraction) -- causes recurring env-runner crashes (see RESULTS.md). + # agent_episodes + per-agent get_return() avoids that global indexing. + for agent_id, agent_ep in episode.agent_episodes.items(): + total = agent_ep.get_return() + if "predator" in agent_id: + predator_total += total + predator_totals.append(total) + group_rewards["predator"].append(total) + elif "prey" in agent_id: + prey_total += total + prey_totals.append(total) + group_rewards["prey"].append(total) + + # Episode summary log + print( + f"Episode {self.num_episodes}: Length: {episode_length} | R={episode_return:.2f} | Global SUM={self.overall_sum_of_rewards:.2f}" + ) + print(f" - Predators: Total = {predator_total:.2f}") + print(f" - Prey: Total = {prey_total:.2f}") + + for group, totals in group_rewards.items(): + print(f" - {group}: Total = {sum(totals):.2f}") + + # Percentile scalars for TensorBoard (appears under Scalars tab) + if predator_totals: + p25, p50, p75 = np.percentile(predator_totals, [25, 50, 75]) + log_value("predator_episode_return_p25", float(p25)) + log_value("predator_episode_return_p50", float(p50)) + log_value("predator_episode_return_p75", float(p75)) + + if prey_totals: + p25, p50, p75 = np.percentile(prey_totals, [25, 50, 75]) + log_value("prey_episode_return_p25", float(p25)) + log_value("prey_episode_return_p50", float(p50)) + log_value("prey_episode_return_p75", float(p75)) + + # Optional: normalize returns by lifetime (if the env exposes lifetime_steps in infos) + infos = self._episode_last_infos(episode) + predator_lifetimes, predator_return_per_life = [], [] + prey_lifetimes, prey_return_per_life = [], [] + for aid, info in infos.items(): + if not isinstance(info, dict): + continue + life = info.get("lifetime_steps") + final_ret = info.get("final_cumulative_reward") + if life is None or final_ret is None or life <= 0: + continue + ret_per_life = float(final_ret) / float(life) + if "predator" in aid: + predator_lifetimes.append(life) + predator_return_per_life.append(ret_per_life) + elif "prey" in aid: + prey_lifetimes.append(life) + prey_return_per_life.append(ret_per_life) + + if predator_lifetimes: + log_value("predator_lifetime_steps_median", float(np.median(predator_lifetimes))) + log_value("predator_return_per_lifetime_mean", float(np.mean(predator_return_per_life))) + if prey_lifetimes: + log_value("prey_lifetime_steps_median", float(np.median(prey_lifetimes))) + log_value("prey_return_per_lifetime_mean", float(np.mean(prey_return_per_life))) + + training_metrics = self._extract_training_metrics(infos, env=env, env_index=env_index, **kwargs) + for metric_name, metric_value in training_metrics.items(): + if isinstance(metric_value, (int, float, np.integer, np.floating)): + log_value(f"eco_evolution/{metric_name}", float(metric_value)) + + # RLlib already emits episode_len_* metrics for TensorBoard; no extra episode-length metrics_logger entry needed here + + def on_episode_step( + self, + *, + episode, + metrics_logger: Optional[MetricsLogger] = None, + env=None, + env_index: int = 0, + **kwargs, + ): + if metrics_logger is None: + return + resolved_env = self._resolve_env(env=env, env_index=env_index, **kwargs) + live_metrics = getattr(resolved_env, "_last_live_culture_metrics", None) + if not isinstance(live_metrics, dict): + return + for metric_name, metric_value in live_metrics.items(): + if isinstance(metric_value, (int, float, np.integer, np.floating)): + metrics_logger.log_value(f"live_culture/{metric_name}", float(metric_value)) + + def on_train_result(self, *, result, **kwargs): + # Add training time metrics + now = time.time() + total_elapsed = now - self.start_time + iter_num = result.get("training_iteration", 1) + iter_time = now - self.last_iteration_time + self.last_iteration_time = now + + result["timing/iter_minutes"] = iter_time / 60.0 + result["timing/avg_minutes_per_iter"] = total_elapsed / 60.0 / iter_num + result["timing/total_hours_elapsed"] = total_elapsed / 3600.0 + # Optional: surface custom metric if available in learner results aggregation + # Ray will automatically aggregate logged metrics across episodes. + + # ---- Compatibility helpers ---- + def _episode_last_infos(self, episode) -> dict: + """ + Return a mapping of agent_id -> last info dict for this episode, with + compatibility across RLlib API changes. + """ + for name in ("get_last_infos", "get_infos"): + if hasattr(episode, name): + try: + infos = getattr(episode, name)() + if isinstance(infos, dict): + return infos + except Exception: + pass + for name in ("last_infos", "infos"): + if hasattr(episode, name): + try: + infos = getattr(episode, name) + if isinstance(infos, dict): + return infos + except Exception: + pass + if hasattr(episode, "_agent_to_last_info"): + try: + mapping = getattr(episode, "_agent_to_last_info") + if isinstance(mapping, dict): + return mapping + except Exception: + pass + return {} + + @staticmethod + def _resolve_env(env=None, env_index: int = 0, **kwargs) -> Any: + """Return the underlying PredPreyGrass env from RLlib wrapper shapes.""" + + def looks_like_metrics_env(obj) -> bool: + return obj is not None and hasattr(obj, "_build_episode_training_metrics") + + def safe_index(value) -> int: + try: + return int(value) + except Exception: + return 0 + + def unwrap(candidate, index: int): + current = candidate + seen = set() + for _ in range(10): + if current is None: + return None + if id(current) in seen: + return None + seen.add(id(current)) + + if looks_like_metrics_env(current): + return current + + if isinstance(current, (list, tuple)): + if not current: + return None + current = current[index] if 0 <= index < len(current) else current[0] + continue + + unwrapped = getattr(current, "unwrapped", None) + if unwrapped is not None and unwrapped is not current: + current = unwrapped + continue + + if hasattr(current, "get_sub_environments"): + try: + sub_envs = current.get_sub_environments() or [] + except Exception: + sub_envs = [] + if sub_envs: + current = sub_envs[index] if 0 <= index < len(sub_envs) else sub_envs[0] + continue + + for attr in ("envs", "_envs"): + sub_envs = getattr(current, attr, None) + if isinstance(sub_envs, (list, tuple)) and sub_envs: + current = sub_envs[index] if 0 <= index < len(sub_envs) else sub_envs[0] + break + else: + sub_envs = None + if sub_envs is not None: + continue + + for attr in ("env", "_env", "vector_env", "_vector_env", "base_env"): + inner = getattr(current, attr, None) + if inner is not None and inner is not current: + current = inner + break + else: + return None + + return None + + index = safe_index(env_index) + for candidate in ( + env, + kwargs.get("env_runner"), + getattr(kwargs.get("env_runner"), "env", None), + kwargs.get("worker"), + getattr(kwargs.get("worker"), "env", None), + kwargs.get("base_env"), + ): + resolved = unwrap(candidate, index) + if resolved is not None: + return resolved + return None + + def _extract_training_metrics(self, infos: dict, env=None, env_index: int = 0, **kwargs) -> dict: + if not isinstance(infos, dict): + infos = {} + else: + all_info = infos.get("__all__") + if isinstance(all_info, dict): + metrics = all_info.get("training_metrics") + if isinstance(metrics, dict): + return metrics + metrics = infos.get("training_metrics") + if isinstance(metrics, dict): + return metrics + + resolved_env = self._resolve_env(env=env, env_index=env_index, **kwargs) + build_metrics = getattr(resolved_env, "_build_episode_training_metrics", None) + if callable(build_metrics): + metrics = build_metrics() + if isinstance(metrics, dict): + return metrics + return {} diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/genome.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/genome.py new file mode 100644 index 000000000..1826dfd0a --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/genome.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +from dataclasses import asdict, dataclass +from typing import Mapping + +import numpy as np + + +@dataclass(frozen=True) +class Genome: + """Heritable dual-inheritance genome for one agent: gene-culture coevolution. + + Two channels, not one: + - `plasticity` (continuous, [0, 1]): the genuinely Darwinian trait. It sets + the per-check probability that this agent's *live* dialect (see + predpreygrass_rllib_env.py's `_apply_cultural_learning`) updates toward + the locally observed majority dialect. Genes here do not encode behavior + directly -- they encode capacity to adopt culture, the literal + Baldwin/dual-inheritance mechanism (Boyd & Richerson). + - `dialect` (categorical, [0, n_dialects)): the agent's *founder* cultural + bias, inherited like any other genome field but immediately overridable + within the agent's own lifetime by `_apply_cultural_learning`. The live, + mutable value lives in `env.agent_live_dialect`, not here -- this field + is only the starting point a newborn is seeded with. + + Policy weights are not part of the genome; movement/hunting behavior + remains within-lifetime PPO adaptation, shared per species as in every + prior eco_evolutionary module. + """ + + plasticity: float + dialect: int + + def to_dict(self) -> dict[str, float]: + return asdict(self) + + +DEFAULT_TRAIT_BOUNDS = { + "plasticity": (0.0, 1.0), +} + +GENOME_FIELD_DEFAULTS: dict[str, float] = { + "plasticity": 0.1, + "dialect": 0, +} + + +def _normal_sample(rng: np.random.Generator, mean: float, std: float, bounds: tuple[float, float]) -> float: + if std <= 0: + value = mean + else: + value = float(rng.normal(mean, std)) + return float(np.clip(value, bounds[0], bounds[1])) + + +def _bounds(config: Mapping, trait: str) -> tuple[float, float]: + configured = config.get("trait_bounds", {}).get(trait, DEFAULT_TRAIT_BOUNDS[trait]) + return float(configured[0]), float(configured[1]) + + +def _n_dialects(config: Mapping) -> int: + return int(config.get("n_dialects", 4)) + + +def founder_genome(policy_group: str, config: Mapping, rng: np.random.Generator) -> Genome: + founder_cfg = config.get("founder_genome", {}).get(policy_group, {}) + plasticity = _normal_sample( + rng, + founder_cfg.get("plasticity_mean", 0.1), + founder_cfg.get("plasticity_std", 0.05), + _bounds(config, "plasticity"), + ) + dialect = int(rng.integers(_n_dialects(config))) + return Genome(plasticity=plasticity, dialect=dialect) + + +def mutate_genome(parent: Genome, config: Mapping, rng: np.random.Generator) -> Genome: + mutation_cfg = config.get("genome_mutation", {}) + rate = float(mutation_cfg.get("rate", 0.0)) + std = float(mutation_cfg.get("std", 0.0)) + + plasticity = parent.plasticity + if rate > 0 and std > 0 and rng.random() < rate: + lo, hi = _bounds(config, "plasticity") + plasticity = float(np.clip(plasticity + rng.normal(0.0, std), lo, hi)) + + dialect_mutation_cfg = config.get("dialect_mutation", {}) + dialect_rate = float(dialect_mutation_cfg.get("rate", 0.0)) + dialect = parent.dialect + if dialect_rate > 0 and rng.random() < dialect_rate: + # Categorical mutation: resample uniformly among all dialects (may land + # back on the same one), same convention as the combinatorial loci + # genome's per-locus mutation in eco_evolutionary_metabolic_code. + dialect = int(rng.integers(_n_dialects(config))) + + return Genome(plasticity=plasticity, dialect=dialect) diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/matplot_renderer.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/matplot_renderer.py new file mode 100644 index 000000000..bf77ab1bb --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/matplot_renderer.py @@ -0,0 +1,371 @@ +import os +import matplotlib.pyplot as plt +from matplotlib.lines import Line2D +import numpy as np + +os.environ["PYGAME_HIDE_SUPPORT_PROMPT"] = "hide" + + +class MatPlotLibRenderer: + """ + A class for visualizing a grid-based environment using Matplotlib. Is used in the RLLib framework. + """ + + # def __init__(self, grid_size, agents, trace_length=5): + def __init__(self, grid_size, agents, trace_length=5, show_gridlines=True, scale=1.0, destination_path=None): + """ + Initialize the visualizer. + + Args: + grid_size (tuple): Size of the grid (rows, cols). + agents (list): List of agent names (e.g., ["predator_0", "prey_0"]). + trace_length (int): Number of steps to retain the movement trace. + destination_path (str): Directory to save plots. If None, plots are not saved automatically. + """ + self.grid_size = grid_size + self.agents = set(agents) + self.trace_length = trace_length + self.agent_traces = {agent: [] for agent in agents} + self.show_gridlines = show_gridlines + self.scale = scale + self.destination_path = destination_path + + # Set up the plot + self.fig, self.ax = plt.subplots(figsize=(6 * self.scale, 6 * self.scale)) + self.ax.set_xlim(-0.5, grid_size[1] - 0.5) + self.ax.set_ylim(-0.5, grid_size[0] - 0.5) + + if self.show_gridlines: + self.ax.set_xticks(range(grid_size[1])) + self.ax.set_yticks(range(grid_size[0])) + self.ax.grid(True, linestyle="--", linewidth=0.5, color="gray") + else: + self.ax.set_xticks([]) + self.ax.set_yticks([]) + self.ax.grid(False) + + self.ax.set_aspect("equal") + + # Flip the y-axis so (0,0) is at the bottom-left + self.ax.invert_yaxis() + + # Store agent markers + self.agent_texts = {} + + # Store trace lines as Line2D objects + self.trace_lines = {} + for agent in agents: + color = "red" if "predator" in agent else "blue" + self.trace_lines[agent] = Line2D([], [], color=color, alpha=0.6, linewidth=1, linestyle="-") + self.ax.add_line(self.trace_lines[agent]) + + # Agent markers + self.predator_marker = "●" + self.prey_marker = "◆" + self.grass_marker = "■" + + def update(self, agent_positions, step): + self.ax.set_title(f"Environment - Step {step}", fontsize=14) + + # Remove old agent markers + for text in self.agent_texts.values(): + text.remove() + self.agent_texts.clear() + + # Get current agent names + current_agents = set(agent_positions.keys()) + + # Remove traces of dead agents + dead_agents = self.agents - current_agents + for agent in dead_agents: + if agent in self.trace_lines: + self.trace_lines[agent].set_data([], []) + if agent in self.agent_traces: + del self.agent_traces[agent] + self.agents = current_agents + + # Update traces for remaining agents + for agent, position in agent_positions.items(): + if "grass" in agent: + # Use the same font scaling for grass + cell_size = min( + self.fig.get_figwidth() * self.fig.dpi / self.grid_size[1], + self.fig.get_figheight() * self.fig.dpi / self.grid_size[0], + ) + font_size = cell_size * 0.6 + + self.agent_texts[agent] = self.ax.text( + position[1], position[0], self.grass_marker, color="green", fontsize=font_size, ha="center", va="center" + ) + continue + + if agent not in self.agent_traces: + self.agent_traces[agent] = [] + if len(self.agent_traces[agent]) >= self.trace_length: + self.agent_traces[agent].pop(0) + self.agent_traces[agent].append(position) + + trace_array = np.array(self.agent_traces[agent]) + if len(trace_array) > 1: + self.trace_lines[agent].set_data(trace_array[:, 1], trace_array[:, 0]) + + # Draw agents + for agent, (x, y) in agent_positions.items(): + if "grass" in agent: + continue + + # Determine marker + marker = self.predator_marker if "predator" in agent else self.prey_marker + + # Determine color + if "predator" in agent: + color = "red" # fallback for classic predator + elif "prey" in agent: + color = "blue" # fallback for classic prey + else: + color = "black" # unknown agent type fallback + + # Scale font to cell size + cell_size = min( + self.fig.get_figwidth() * self.fig.dpi / self.grid_size[1], + self.fig.get_figheight() * self.fig.dpi / self.grid_size[0], + ) + font_size = cell_size * 0.6 # 60% of the cell size for padding + + self.agent_texts[agent] = self.ax.text(y, x, marker, color=color, fontsize=font_size, ha="center", va="center") + + plt.draw() + plt.pause(0.01) + + def save_frame(self, step, prefix="grid_step"): + """ + Save the current grid as an image file with the step number in the filename. + """ + if self.destination_path: + save_dir = os.path.join(self.destination_path, "grid_step_plots") + os.makedirs(save_dir, exist_ok=True) + filename = f"{prefix}_{step:04d}.png" + filepath = os.path.join(save_dir, filename) + self.fig.savefig(filepath) + # print(f"Saved grid frame: {filepath}") + + def plot(self, save_name="grid_summary.png"): + """Save the final plot summary.""" + if self.destination_path: + os.makedirs(self.destination_path, exist_ok=True) + path = os.path.join(self.destination_path, save_name) + self.fig.savefig(path) + # print(f"Saved final grid plot: {path}") + else: + # Display the population chart plot to the user + plt.show() + + def close(self): + """Close the visualization.""" + plt.close(self.fig) + + +class EvolutionVisualizer: + def __init__(self): + self.role_counts_dict = {"predator": [], "prey": []} + + def record_counts(self, active_agent_names): + self.role_counts_dict["predator"].append(sum(1 for name in active_agent_names if "predator" in name)) + self.role_counts_dict["prey"].append(sum(1 for name in active_agent_names if "prey" in name)) + + def plot(self): + role_counts_dict = self.role_counts_dict + plt.figure(figsize=(8, 5)) + + for label, counts in role_counts_dict.items(): + plt.plot(counts, label=label.replace("_", " ").capitalize()) + plt.xlabel("Step") + plt.ylabel("Number of Agents") + plt.title("Agent Population") + plt.legend() + plt.grid(True) + + plt.tight_layout() + plt.show() + + +class AverageAgeVisualizer: + def __init__(self): + self.history = {"predator": [], "prey": []} + + def record(self, agent_internal_ids, agent_ages, agent_positions): + # Initialize sum and count per group + sums = {k: 0 for k in self.history} + counts = {k: 0 for k in self.history} + + for agent_id, internal_id in agent_internal_ids.items(): + if agent_id not in agent_positions: + continue # Only include active agents + + age = agent_ages.get(internal_id, 0) + + if "prey" in agent_id: + sums["prey"] += age + counts["prey"] += 1 + elif "predator" in agent_id: + sums["predator"] += age + counts["predator"] += 1 + + # Compute and record averages (0 if no agents) + for group in self.history: + avg = sums[group] / counts[group] if counts[group] > 0 else 0 + self.history[group].append(avg) + + def plot(self): + plt.figure(figsize=(10, 5)) + for label, values in self.history.items(): + plt.plot(values, label=label.replace("_", " ").capitalize()) + plt.xlabel("Step") + plt.ylabel("Average Age") + plt.title("Average Age of Agent Groups Over Time") + plt.legend() + plt.grid(True) + plt.tight_layout() + plt.show() + + +class PopulationChart: + def __init__(self, destination_path=None): + self.destination_path = destination_path + self.time_steps = [] + self.predator_counts = [] + self.prey_counts = [] + + def record(self, step, agents): + self.time_steps.append(step) + self.predator_counts.append(sum(1 for a in agents if "predator" in a)) + self.prey_counts.append(sum(1 for a in agents if "prey" in a)) + + def plot(self): + plt.figure(figsize=(10, 5)) + plt.plot(self.time_steps, self.predator_counts, label="Predators", color="red") + plt.plot(self.time_steps, self.prey_counts, label="Prey", color="blue") + plt.xlabel("Time Step") + plt.ylabel("Number of Agents") + plt.title("Agent Population Over Time") + plt.legend() + plt.grid(True) + plt.tight_layout() + + if self.destination_path: + plot_dir = os.path.join(self.destination_path, "summary_plots") + os.makedirs(plot_dir, exist_ok=True) + filepath = os.path.join(plot_dir, "population_chart.png") + plt.savefig(filepath) + # print(f"Population chart saved to: {filepath}") + else: + plt.show() + + +class CombinedEvolutionVisualizer: + def __init__(self, destination_path=None, timestamp=None, destination_filename="summary_plots", run_nr=None): + self.destination_path = destination_path + self.destination_filename = destination_filename + self.timestamp = timestamp + self.run_nr = run_nr + # Population counts + self.time_steps = [] + self.predator_counts = [] + self.prey_counts = [] + self.energy_by_type_series = [] + + def record(self, agent_ids): + step = len(self.time_steps) + self.time_steps.append(step) + self.predator_counts.append(sum(1 for a in agent_ids if "predator" in a)) + self.prey_counts.append(sum(1 for a in agent_ids if "prey" in a)) + + def record_energy(self, energy_dict): + """Stores aggregate predator, prey, and grass energy for each step.""" + self.energy_by_type_series.append(energy_dict.copy()) + + def plot(self): + steps = self.time_steps + n_charts = 1 + if self.energy_by_type_series: + n_charts += 1 + plt.figure(figsize=(8 * n_charts, 6)) + + # 1. Total predator and prey count + plt.subplot(1, n_charts, 1) + plt.plot(steps, self.predator_counts, label="Predators", color="red", linewidth=2) + plt.plot(steps, self.prey_counts, label="Prey", color="blue", linewidth=2) + plt.title("Agent Population") + plt.xlabel("Step") + plt.ylabel("Count") + plt.legend() + plt.grid(True) + + # 2. Aggregate energy by role + if self.energy_by_type_series: + plt.subplot(1, n_charts, 2) + steps = list(range(len(self.energy_by_type_series))) + energy_keys = ["predator", "prey", "grass"] + color_map = {"predator": "red", "prey": "blue", "grass": "green", "total": "black"} + for k in energy_keys: + series = [entry[k] for entry in self.energy_by_type_series] + plt.plot(steps, series, label=k.capitalize(), linewidth=2, color=color_map[k]) + # Add total energy line + total_series = [sum(entry[k] for k in energy_keys) for entry in self.energy_by_type_series] + plt.plot(steps, total_series, label="Total", linestyle="--", linewidth=2, color=color_map["total"]) + + plt.title("Total Energy per Agent Type") + plt.xlabel("Step") + plt.ylabel("Energy") + plt.legend() + plt.grid(True) + + plt.tight_layout() + + if self.destination_path: + os.makedirs(os.path.join(self.destination_path, "summary_plots"), exist_ok=True) + path = os.path.join(self.destination_path, "summary_plots", "evolution_summary_" + str(self.run_nr) + ".png") + plt.savefig(path) + # plt.show() + else: + plt.show() + + +class PreyDeathCauseVisualizer: + def __init__(self, destination_path=None, timestamp=None, destination_filename="summary_plots"): + self.timestamp = timestamp + self.destination_path = destination_path + self.destination_filename = destination_filename + self.time_steps = [] + self.starved_ratio = [] + self.eaten_ratio = [] + + def record(self, death_cause_prey): + step = len(self.time_steps) + self.time_steps.append(step) + starved = sum(1 for cause in death_cause_prey.values() if cause == "starved") + eaten = sum(1 for cause in death_cause_prey.values() if cause == "eaten") + total = starved + eaten + self.starved_ratio.append(starved / total if total > 0 else 0) + self.eaten_ratio.append(eaten / total if total > 0 else 0) + + def plot(self): + plt.figure(figsize=(8, 5)) + plt.plot(self.time_steps, [s * 100 for s in self.starved_ratio], label="Starved Prey %", color="blue", linewidth=2) + # plt.plot(self.time_steps, [e * 100 for e in self.eaten_ratio], label="Eaten Prey %", color="black", linestyle="--", linewidth=2) + plt.title("Prey Death Cause Starvation Relative") + plt.xlabel("Step") + plt.ylabel("Percentage (%)") + # plt.ylim(0, 100) + plt.legend() + plt.grid(True) + + if self.destination_path: + os.makedirs(os.path.join(self.destination_path, self.destination_filename), exist_ok=True) + path = os.path.join( + self.destination_path, self.destination_filename, "prey_death_cause_plot_" + str(self.timestamp) + ".png" + ) + plt.savefig(path) + plt.show() + else: + plt.show() diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/networks.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/networks.py new file mode 100644 index 000000000..998e0de2c --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/networks.py @@ -0,0 +1,116 @@ + +from ray.rllib.core.rl_module import RLModuleSpec +from ray.rllib.core.rl_module.multi_rl_module import MultiRLModuleSpec +from ray.rllib.algorithms.ppo.torch.default_ppo_torch_rl_module import DefaultPPOTorchRLModule + + +def build_module_spec(obs_space, act_space, policy_name: str = None): + """ + Build an RLModuleSpec whose conv depth L matches the observation window size H + so that the receptive field RF = 1 + 2L equals H. + Also widens the first FC layer for large action spaces (>20 actions). + """ + # obs_space is a Box with shape (C, H, W) + # C = number of channels (predators, prey, grass -> usually 3) + # H = height of the square observation window (e.g. 7 for predators, 9 for prey) + # W = width of the square observation window (equal to H here) + C, H, W = obs_space.shape + + # We assert the window is square and odd-sized. + # Odd size is important because the agent is always centered in its window. + assert H == W and H % 2 == 1, "Expected odd square obs windows (e.g., 7x7, 9x9)." + + # Receptive field math: + # Each 3x3 stride-1 conv layer expands the receptive field by +2. + # General formula: RF = 1 + 2 * L + # To cover the full observation window (size H), we solve: + # H = 1 + 2L → L = (H - 1) // 2 + # Example: H=7 → L=3 conv layers (RF=7), H=9 → L=4 conv layers (RF=9). + L = (H - 1) // 2 + + # Channel schedule: + # We start small (16 filters), then increase (32, 64). + # If more layers are needed (e.g., prey with 9x9), we keep adding 64-channel layers. + base_channels = [16, 32, 64] + if L <= len(base_channels): + channels = base_channels[:L] + else: + channels = base_channels + [64] * (L - len(base_channels)) + + # Assemble conv_filters list for RLlib (format: [num_filters, [kernel, kernel], stride]) + conv_filters = [[c, [3, 3], 1] for c in channels] + + # Adjust the fully-connected (FC) hidden sizes based on the action space. + # If the agent has many actions (e.g., prey with 25 moves), we give it a wider first FC layer + # so it has more capacity to rank actions effectively. + num_actions = act_space.n if hasattr(act_space, "n") else None + if num_actions is not None and num_actions > 20: + fcnet_hiddens = [384, 256] + head_note = "wide" + else: + fcnet_hiddens = [256, 256] + head_note = "standard" + + # ---- Debug/trace log (once per policy) ---- + # Example: [MODEL] prey → obs CxHxW=4x9x9, L=4 (RF=9), conv=[16,32,64,64], actions=25, head=wide + if policy_name is not None: + rf = 1 + 2 * L + conv_str = ",".join(str(c) for c in channels) + print( + f"[MODEL] {policy_name} → obs CxHxW={C}x{H}x{W}, " + f"L={L} (RF={rf}), conv=[{conv_str}], " + f"actions={num_actions}, head={head_note}" + ) + + + return RLModuleSpec( + module_class=DefaultPPOTorchRLModule, + observation_space=obs_space, + action_space=act_space, + inference_only=False, + model_config={ + "conv_filters": conv_filters, + "fcnet_hiddens": fcnet_hiddens, + "fcnet_activation": "relu", + }, + ) + +def build_multi_module_spec( + obs_spaces_by_policy: dict, + act_spaces_by_policy: dict, +) -> MultiRLModuleSpec: + """ + Build a MultiRLModuleSpec for multiple policies. + + Args: + obs_spaces_by_policy: dict mapping policy_id -> observation_space + act_spaces_by_policy: dict mapping policy_id -> action_space + + Returns: + MultiRLModuleSpec containing an RLModuleSpec per policy, using + DefaultPPOTorchRLModule with conv/FC settings chosen by build_module_spec(). + + Notes: + - Keys of obs_spaces_by_policy and act_spaces_by_policy must match. + - This function prints one model summary line per policy (via build_module_spec). + """ + # Ensure both dicts have the same policy IDs + obs_keys = set(obs_spaces_by_policy.keys()) + act_keys = set(act_spaces_by_policy.keys()) + if obs_keys != act_keys: + missing_in_act = sorted(obs_keys - act_keys) + missing_in_obs = sorted(act_keys - obs_keys) + raise ValueError( + f"Policy key mismatch. " + f"Missing in act: {missing_in_act}; Missing in obs: {missing_in_obs}" + ) + + rl_module_specs = {} + for policy_id in obs_keys: + rl_module_specs[policy_id] = build_module_spec( + obs_spaces_by_policy[policy_id], + act_spaces_by_policy[policy_id], + policy_name=policy_id, + ) + + return MultiRLModuleSpec(rl_module_specs=rl_module_specs) diff --git a/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/pygame_grid_renderer_rllib.py b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/pygame_grid_renderer_rllib.py new file mode 100644 index 000000000..52628d4c7 --- /dev/null +++ b/predpreygrass/evolutionary/eco_evolutionary_cultural_plasticity/utils/pygame_grid_renderer_rllib.py @@ -0,0 +1,675 @@ +import pygame +from dataclasses import dataclass +from typing import Any, Callable + + +@dataclass +class GuiStyle: + margin_left: int = 10 + margin_top: int = 10 + margin_right: int = 340 + margin_bottom: int = 10 + legend_spacing: int = 30 + legend_font_size: int = 28 + legend_circle_radius: int = 10 + legend_square_size: int = 20 + tooltip_font_size: int = 28 + tooltip_padding: int = 4 + + predator_color: tuple = (255, 0, 0) + prey_color: tuple = (0, 0, 255) + grass_color: tuple = (0, 128, 0) + grid_color: tuple = (200, 200, 200) + background_color: tuple = (255, 255, 255) + halo_reproduction_color: tuple = (255, 0, 0) + halo_eating_color: tuple = (0, 128, 0) # Bright green + halo_reproduction_thickness: int = 3 + halo_eating_thickness: int = 3 + + +class PyGameRenderer: + def __init__( + self, + grid_size, + cell_size=32, + enable_speed_slider=True, + enable_tooltips=True, + max_steps=None, + predator_obs_range=None, + prey_obs_range=None, + show_fov=True, + fov_alpha=70, + fov_agents=None, + ): + self.grid_size = grid_size + self.cell_size = cell_size + self.enable_speed_slider = enable_speed_slider + self.enable_tooltips = enable_tooltips + self.gui_style = GuiStyle() + + # Observation/FOV overlay configuration + self.predator_obs_range = predator_obs_range + self.prey_obs_range = prey_obs_range + self.show_fov = show_fov + self.fov_alpha = fov_alpha # 0-255 + # Optional: restrict FOV overlay to specific agent IDs (exact match). If None, draw for all. + self.fov_agents = set(fov_agents) if fov_agents is not None else None + # Per-species FOV colors (RGBA) + self.fov_color_predator = (255, 0, 0, self.fov_alpha) # Red with alpha + self.fov_color_prey = (0, 0, 255, self.fov_alpha) # Blue with alpha + + window_width = self.gui_style.margin_left + grid_size[0] * cell_size + self.gui_style.margin_right + window_height = self.gui_style.margin_top + grid_size[1] * cell_size + self.gui_style.margin_bottom + pygame.init() + self.screen = pygame.display.set_mode((window_width, window_height)) + pygame.display.set_caption("PredPreyGrass Live Viewer") + + self.font = pygame.font.SysFont(None, int(cell_size * 0.5)) + self.font_legend = pygame.font.SysFont(None, self.gui_style.legend_font_size) + + self.tooltip_font = pygame.font.SysFont(None, self.gui_style.tooltip_font_size) + + self.reference_energy_predator = 10.0 # referenc wrt size of predators + self.reference_energy_prey = 3.0 + self.reference_energy_grass = 2.0 + + self.halo_trigger_fraction = 0.9 + self.predator_creation_energy_threshold = 12.0 + self.prey_creation_energy_threshold = 8.0 + + self.previous_agent_energies = {} + self.population_history_steps = [] + self.population_history_predators = [] + self.population_history_prey = [] + self.population_history_max_length = 1000 + + self.target_fps = 10 # Default FPS + self.slider_rect: pygame.Rect | None = None # Will be defined in _draw_legend() + self.slider_max_fps = 60 + self.population_history_max_length = max_steps if max_steps else 1000 + + self.grid_surface = pygame.Surface((window_width, window_height), pygame.SRCALPHA) + for x in range(self.grid_size[0]): + for y in range(self.grid_size[1]): + rect = pygame.Rect( + self.gui_style.margin_left + x * self.cell_size, + self.gui_style.margin_top + y * self.cell_size, + self.cell_size, + self.cell_size, + ) + pygame.draw.rect(self.grid_surface, self.gui_style.grid_color, rect, 1) + + def update(self, grass_positions, grass_energies=None, step=0, agents_just_ate=None, per_step_agent_data=None, dead_prey=None): + step_data = per_step_agent_data[step - 1] + if agents_just_ate is None: + agents_just_ate = set() + if dead_prey is None: + dead_prey = set() + + self.screen.fill(self.gui_style.background_color) + + # Count predator and prey population using step_data + num_predators = num_prey = 0 + for agent_id in step_data: + if "predator" in agent_id: + num_predators += 1 + elif "prey" in agent_id: + num_prey += 1 + + self.population_history_steps.append(step) + self.population_history_predators.append(num_predators) + self.population_history_prey.append(num_prey) + + # Keep history length bounded + if len(self.population_history_steps) > self.population_history_max_length: + self.population_history_steps.pop(0) + self.population_history_predators.pop(0) + self.population_history_prey.pop(0) + + self._draw_grid() + if self.show_fov and (self.predator_obs_range or self.prey_obs_range): + self._draw_fov_overlays(step_data) + self._draw_grass(grass_positions, grass_energies) + self._draw_agents(step_data, agents_just_ate, dead_prey) + self._draw_legend(step, step_data) + if self.enable_tooltips: + self._draw_tooltip(step_data, grass_positions, grass_energies) + + pygame.display.set_caption(f"PredPreyGrass Live Viewer — Step {step}") + pygame.display.flip() + + def _draw_grid(self): + self.screen.blit(self.grid_surface, (0, 0)) + + def _draw_grass(self, grass_positions, grass_energies): + for grass_id, pos in grass_positions.items(): + x_pix = self.gui_style.margin_left + pos[0] * self.cell_size + self.cell_size // 2 + y_pix = self.gui_style.margin_top + pos[1] * self.cell_size + self.cell_size // 2 + color = self.gui_style.grass_color + energy = grass_energies.get(grass_id, 0) if grass_energies else 0 + size_factor = min(energy / self.reference_energy_grass, 1.0) + base_rect_size = self.cell_size * 0.8 + rect_size = base_rect_size * size_factor + rect = pygame.Rect(x_pix - rect_size // 2, y_pix - rect_size // 2, rect_size, rect_size) + pygame.draw.rect(self.screen, color, rect) + + def _draw_fov_overlays(self, step_data): + """Draw transparent FOV overlays for each agent.""" + if self.fov_alpha <= 0: + return + # Create a small surface for blending + cs = self.cell_size + ml = self.gui_style.margin_left + mt = self.gui_style.margin_top + + def _range_for_agent(aid): + if "predator" in aid: + return self.predator_obs_range + return self.prey_obs_range + + for agent_id, data in step_data.items(): + if self.fov_agents is not None and agent_id not in self.fov_agents: + continue + obs_r = _range_for_agent(agent_id) + if not obs_r or obs_r <= 0: + continue + pos = tuple(map(int, data["position"])) + offset = (obs_r - 1) // 2 + x0 = max(0, pos[0] - offset) + y0 = max(0, pos[1] - offset) + x1 = min(self.grid_size[0] - 1, pos[0] + offset) + y1 = min(self.grid_size[1] - 1, pos[1] + offset) + + if "predator" in agent_id: + base_color = self.fov_color_predator + else: + base_color = self.fov_color_prey + + px = ml + x0 * cs + py = mt + y0 * cs + w = (x1 - x0 + 1) * cs + h = (y1 - y0 + 1) * cs + surf = pygame.Surface((w, h), pygame.SRCALPHA) + surf.fill(base_color) + self.screen.blit(surf, (px, py)) + + def _draw_agents(self, step_data, agents_just_ate, dead_prey): + for agent_id, agent in step_data.items(): + pos = tuple(map(int, agent["position"])) + energy = agent["energy"] + + # Convert grid position to pixel coordinates + x_pix = self.gui_style.margin_left + pos[0] * self.cell_size + self.cell_size // 2 + y_pix = self.gui_style.margin_top + pos[1] * self.cell_size + self.cell_size // 2 + + if "predator" in agent_id: + color = self.gui_style.predator_color # Default predator color + reference_energy = self.reference_energy_predator + + elif "prey" in agent_id: + # Dead prey (carcass-like) rendered in a distinct color + if agent_id in dead_prey: + color = (128, 128, 128) # Gray for dead prey + else: + color = self.gui_style.prey_color # Default prey color + reference_energy = self.reference_energy_prey + + size_factor = min(energy / reference_energy, 1.0) + base_radius = self.cell_size // 2 - 2 + radius = int(base_radius * size_factor) + + pygame.draw.circle(self.screen, color, (x_pix, y_pix), max(radius, 2)) + + # Eating halo (green ring) + if agent_id in agents_just_ate: + pygame.draw.circle( + self.screen, + self.gui_style.halo_eating_color, + (x_pix, y_pix), + max(radius + 5, 6), + width=self.gui_style.halo_eating_thickness, + ) + + # Reproduction halo (red ring) + if "energy_reproduction" in agent and agent["energy_reproduction"] < 0: + pygame.draw.circle( + self.screen, + self.gui_style.halo_reproduction_color, + (x_pix, y_pix), + max(radius + 5, 6), + width=self.gui_style.halo_reproduction_thickness, + ) + + def _draw_legend(self, step, step_data): + x = self.gui_style.margin_left + self.grid_size[0] * self.cell_size + 20 + y = self.gui_style.margin_top + 10 + + y = self._draw_legend_step_counter(x, y, step) + + y = self._draw_legend_agents(x, y, step_data) + + y = self._draw_legend_environment_elements(x, y) + + if self.enable_speed_slider: + y = self._draw_legend_type_slider(x, y) + + y = self._draw_legend_population_chart(x, y) + + def _draw_legend_step_counter(self, x, y, step): + spacing = self.gui_style.legend_spacing + font_large = self.font_legend + step_label_surface = font_large.render("Step:", True, (0, 0, 0)) + self.screen.blit(step_label_surface, (x, y)) + + label_width = step_label_surface.get_width() + step_number_surface = font_large.render(f"{step}", True, (255, 0, 0)) + self.screen.blit(step_number_surface, (x + label_width + 5, y)) # +5 pixels spacing + + return y + spacing + + def _draw_legend_agents(self, x, y, step_data): + spacing = self.gui_style.legend_spacing + r = self.gui_style.legend_circle_radius + font = self.tooltip_font + font_large = self.font_legend + + # Draw legend title + title_surface = font_large.render("Agent size depends on energy", True, (0, 0, 0)) + self.screen.blit(title_surface, (x, y)) + y += spacing + + pygame.draw.circle(self.screen, self.gui_style.predator_color, (x + r, y + r), r) + self.screen.blit(font.render("Predator", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + pygame.draw.circle(self.screen, self.gui_style.prey_color, (x + r, y + r), r) + self.screen.blit(font.render("Prey", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + # Reproduction halo + pygame.draw.circle( + self.screen, + self.gui_style.halo_reproduction_color, + (x + r, y + r), + r + 2, + width=self.gui_style.halo_reproduction_thickness, + ) + self.screen.blit(font.render("Reproduction halo", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + # Eating halo + pygame.draw.circle( + self.screen, self.gui_style.halo_eating_color, (x + r, y + r), r + 2, width=self.gui_style.halo_eating_thickness + ) + self.screen.blit(font.render("Eating halo", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + return y + + def _draw_legend_environment_elements(self, x, y): + spacing = self.gui_style.legend_spacing + r = self.gui_style.legend_circle_radius + s = self.gui_style.legend_square_size + font = self.tooltip_font + + # Grass + pygame.draw.rect(self.screen, self.gui_style.grass_color, pygame.Rect(x + r - s // 2, y + r - s // 2, s, s)) + self.screen.blit(font.render("Grass", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + if self.show_fov and (self.predator_obs_range or self.prey_obs_range): + # Predator FOV legend (with alpha blending) + fov_pred_rect = pygame.Surface((s, s), pygame.SRCALPHA) + fov_pred_rect.fill(self.fov_color_predator) + self.screen.blit(fov_pred_rect, (x + r - s // 2, y + r - s // 2)) + self.screen.blit(font.render("Predator_0 Field Of Vision", True, (0, 0, 0)), (x + 30, y)) + y += spacing + # Prey FOV legend (with alpha blending) + fov_prey_rect = pygame.Surface((s, s), pygame.SRCALPHA) + fov_prey_rect.fill(self.fov_color_prey) + self.screen.blit(fov_prey_rect, (x + r - s // 2, y + r - s // 2)) + self.screen.blit(font.render("Prey_0 Field Of Vision", True, (0, 0, 0)), (x + 30, y)) + y += spacing + + return y + + def _draw_legend_type_slider(self, x, y): + spacing = self.gui_style.legend_spacing + font = pygame.font.SysFont(None, 24) + + y += spacing + + slider_label_surface = font.render("Speed (steps/sec)", True, (0, 0, 0)) + self.screen.blit(slider_label_surface, (x, y)) + + y += spacing + + slider_x = x + slider_y = y + slider_width = 200 + slider_height = 20 + + pygame.draw.rect(self.screen, (180, 180, 180), pygame.Rect(slider_x, slider_y, slider_width, slider_height)) + pygame.draw.rect(self.screen, (0, 0, 0), pygame.Rect(slider_x, slider_y, slider_width, slider_height), 1) + + slider_max_fps = self.slider_max_fps + ratio = (self.target_fps - 1) / (slider_max_fps - 1) + handle_x = slider_x + int(ratio * slider_width) + handle_y = slider_y + slider_height // 2 + + pygame.draw.circle(self.screen, (50, 50, 250), (handle_x, handle_y), 16) + + fps_surface = font.render(f"{self.target_fps} FPS", True, (0, 0, 0)) + self.screen.blit(fps_surface, (slider_x + slider_width + 15, slider_y - 8)) + + self.slider_rect = pygame.Rect(slider_x, slider_y, slider_width, slider_height) + + return y + spacing # Return new Y position for chart continuation + + def _draw_legend_population_chart(self, x, y): + chart_width = 260 + chart_height = 100 + chart_x = x + 30 + chart_y = y + 40 + spacing = self.gui_style.legend_spacing + + # Chart title + font_small = pygame.font.SysFont(None, int(self.gui_style.tooltip_font_size * 0.8), bold=False) + title_surface = font_small.render("Total predator and prey population", True, (0, 0, 0)) + title_x = chart_x + chart_width // 2 - title_surface.get_width() // 2 + title_y = chart_y - spacing // 2 - 10 + self.screen.blit(title_surface, (title_x, title_y)) + + # Chart box + pygame.draw.rect(self.screen, (230, 230, 230), pygame.Rect(chart_x, chart_y, chart_width, chart_height)) + pygame.draw.rect(self.screen, (0, 0, 0), pygame.Rect(chart_x, chart_y, chart_width, chart_height), 1) + + # Y axis + max_agents = max(max(self.population_history_predators, default=1), max(self.population_history_prey, default=1), 1) + num_ticks = 5 + label_x = chart_x - 5 + font_small = pygame.font.SysFont(None, int(self.gui_style.tooltip_font_size * 0.8), bold=False) + + for i in range(num_ticks + 1): + value = int(i / num_ticks * max_agents) + y_pos = chart_y + chart_height - int(i / num_ticks * chart_height) + label_surface = font_small.render(f"{value}", True, (0, 0, 0)) + label_width = label_surface.get_width() + self.screen.blit(label_surface, (label_x - label_width, y_pos - label_surface.get_height() // 2)) + + # X ticks + max_len = self.population_history_max_length + step = max_len // 5 + x_tick_labels = list(range(0, max_len + 1, step)) + num_xticks = len(x_tick_labels) + for i, label_value in enumerate(x_tick_labels): + x_pos = chart_x + int(i / (num_xticks - 1) * chart_width) + y_pos = chart_y + chart_height + 5 + label_surface = font_small.render(f"{label_value}", True, (0, 0, 0)) + label_width = label_surface.get_width() + self.screen.blit(label_surface, (x_pos - label_width // 2, y_pos)) + + # Draw line chart + if self.population_history_steps: + for i in range(1, len(self.population_history_steps)): + x1 = chart_x + int((i - 1) / self.population_history_max_length * chart_width) + x2 = chart_x + int(i / self.population_history_max_length * chart_width) + y1p = chart_y + chart_height - int(self.population_history_predators[i - 1] / max_agents * chart_height) + y2p = chart_y + chart_height - int(self.population_history_predators[i] / max_agents * chart_height) + pygame.draw.line(self.screen, self.gui_style.predator_color, (x1, y1p), (x2, y2p), 2) + y1pr = chart_y + chart_height - int(self.population_history_prey[i - 1] / max_agents * chart_height) + y2pr = chart_y + chart_height - int(self.population_history_prey[i] / max_agents * chart_height) + pygame.draw.line(self.screen, self.gui_style.prey_color, (x1, y1pr), (x2, y2pr), 2) + + return chart_y + chart_height + self.gui_style.legend_spacing + + def _draw_tooltip(self, step_data, grass_positions, grass_energies): + mouse_x, mouse_y = pygame.mouse.get_pos() + grid_x = (mouse_x - self.gui_style.margin_left) // self.cell_size + grid_y = (mouse_y - self.gui_style.margin_top) // self.cell_size + + # Early exit if mouse is outside grid area + if not (0 <= grid_x < self.grid_size[0]) or not (0 <= grid_y < self.grid_size[1]): + return + + hovered_entity = None + hovered_energy = 0.0 + + # Check for hovered agent + for agent_id, data in step_data.items(): + pos = tuple(map(int, data["position"])) + if pos == (grid_x, grid_y): + hovered_entity = agent_id + hovered_energy = data["energy"] + break + + # Check for hovered grass (only if no agent found) + if not hovered_entity: + for grass_id, pos in grass_positions.items(): + if pos == (grid_x, grid_y): + hovered_entity = grass_id + hovered_energy = grass_energies.get(grass_id, 0) if grass_energies else 0 + break + + if hovered_entity: + font = self.tooltip_font + lines = [] + + # First line: agent or grass ID + lines.append(("ID", hovered_entity)) + + if hovered_entity in step_data: + agent = step_data[hovered_entity] + + age = agent.get("age") + if age is not None: + lines.append(("Age", f"{age:>6}")) + + # Offspring count from the current live agent only + offspring = agent.get("offspring_count") + if offspring is not None: + lines.append(("Offspring", f"{offspring:>6}")) + + if "offspring_ids" in agent and agent["offspring_ids"]: + for i, uid in enumerate(agent["offspring_ids"]): + lines.append((f"Child {i+1}", uid)) + + lines.append(("Energy", f"{hovered_energy:+6.2f}")) + + # Energy deltas + for key, label in [ + ("energy_decay", "-Decay"), + ("energy_movement", "-Movement"), + ("energy_eating", "-Eating"), + ("energy_reproduction", "-Reproduction"), + ]: + if key in agent: + delta = agent[key] + lines.append((label, f"{delta:+6.2f}")) + + # Add implicit energy from previous step for debugging + if all(k in agent for k in ("energy_decay", "energy_movement", "energy_eating", "energy_reproduction")): + decay = agent["energy_decay"] + move = agent["energy_movement"] + eat = agent["energy_eating"] + repro = agent["energy_reproduction"] + energy_prev = hovered_energy - decay - move - eat - repro + lines.append(("Energy_prev", f"{energy_prev:6.2f}")) + else: + lines.append(("Energy", f"{hovered_energy:>6.2f}")) + + # Measure width/height + label_width = max(font.size(label)[0] for label, _ in lines if label) + value_width = max(font.size(value)[0] for _, value in lines if value) + line_height = font.get_height() + padding = self.gui_style.tooltip_padding + width = label_width + value_width + 12 + height = len(lines) * line_height + + tooltip_x = mouse_x + 10 + tooltip_y = mouse_y + 10 + + bg_rect = pygame.Rect(tooltip_x - padding, tooltip_y - padding, width + 2 * padding, height + 2 * padding) + pygame.draw.rect(self.screen, (255, 255, 200), bg_rect) + pygame.draw.rect(self.screen, (0, 0, 0), bg_rect, 1) + + # Render lines + y_offset = tooltip_y + for label, value in lines: + if value is None: + text_surface = font.render(label, True, (0, 0, 0)) + self.screen.blit(text_surface, (tooltip_x, y_offset)) + else: + label_surface = font.render(f"{label}:", True, (0, 0, 0)) + value_surface = font.render(value, True, (0, 0, 0)) + self.screen.blit(label_surface, (tooltip_x, y_offset)) + self.screen.blit(value_surface, (tooltip_x + label_width + 12, y_offset)) + y_offset += line_height + + def close(self): + pygame.quit() + + +class ViewerControlHelper: + """ + Helper class to manage viewer control: + - Pause / Play (SPACE) + - Single Step (RIGHT arrow) + - Quit (window close or ESC) + + Usage: + + control = ViewerControlHelper() + + for step in ...: + control.handle_events() + + if not control.paused or control.step_once: + # env.step() + # viewer.update() + control.step_once = False + else: + # viewer.update() + pygame.time.wait(50) + """ + + def __init__(self, initial_paused=False): + self.paused = initial_paused + self.step_once = False + self.step_backward = False + self.fps_slider_rect: pygame.Rect | None = None + self.fps_slider_update_fn: Callable[[int], None] | None = None + self.visualizer: Any | None = None + self.is_dragging_slider = False + + def handle_events(self): + for event in pygame.event.get(): + if event.type == pygame.QUIT: + print("[ViewerControl] Quit detected — exiting.") + pygame.quit() + raise SystemExit(0) + + elif event.type == pygame.KEYDOWN: + if event.key == pygame.K_ESCAPE: + print("[ViewerControl] ESC pressed — exiting.") + pygame.quit() + raise SystemExit(0) + + elif event.key == pygame.K_SPACE: + self.paused = not self.paused + print(f"[ViewerControl] Pause {'ON' if self.paused else 'OFF'}") + + elif event.key == pygame.K_RIGHT: + self.paused = True + self.step_once = True + print("[ViewerControl] Single Step") + + elif event.key == pygame.K_LEFT: + self.paused = True + self.step_backward = True + print("[ViewerControl] Step Backward") + elif event.type == pygame.MOUSEBUTTONDOWN: + if event.button == 1: + if self.fps_slider_rect and self.fps_slider_rect.collidepoint(event.pos): + self.is_dragging_slider = True + self._update_fps_slider(event.pos[0]) + + elif event.type == pygame.MOUSEMOTION: + if self.is_dragging_slider: + self._update_fps_slider(event.pos[0]) + + elif event.type == pygame.MOUSEBUTTONUP: + if event.button == 1 and self.is_dragging_slider: + self.is_dragging_slider = False + + def _update_fps_slider(self, slider_x): + if self.fps_slider_rect is None or self.fps_slider_update_fn is None or self.visualizer is None: + return + slider_max_fps = self.visualizer.slider_max_fps + slider_start_x = self.fps_slider_rect.left + slider_width = self.fps_slider_rect.width + ratio = (slider_x - slider_start_x) / slider_width + ratio = min(max(ratio, 0.0), 1.0) + new_fps = int(1 + ratio * (slider_max_fps - 1)) + self.fps_slider_update_fn(new_fps) + print(f"[ViewerControl] Target FPS set to {new_fps}") + + +class LoopControlHelper: + """ + LoopControlHelper + + Purpose: + - Provide a small helper to manage 'simulation terminated' state safely. + - Avoid accidental termination of the main loop when paused. + - Provide 'should_step()' logic for consistent handling of pause/play/step. + + Why: + - Without this helper, many users accidentally update 'done' on every loop iteration. + → Result: when you pause, step, or hover, your loop can exit because 'done' was True from a previous step. + - The correct pattern is to only update 'simulation_terminated' **after env.step()**, not in every loop. + - This helper encapsulates that safe pattern. + - It also gives a reusable 'should_step()' function → makes the loop structure uniform across scripts. + + Typical usage: + + control = ViewerControlHelper() + loop_helper = LoopControlHelper() + + while not loop_helper.simulation_terminated: + control.handle_events() + + if loop_helper.should_step(control): + obs, rewards, terminations, truncations, info = env.step(action_dict) + + live_viewer.update(...) + loop_helper.update_simulation_terminated(terminations, truncations) + + control.step_once = False + ... + + else: + live_viewer.update(...) + pygame.time.wait(50) + + Summary: + - This helper is very small, but ensures your viewer loops behave safely. + - Prevents bugs when combining pause/play/step with 'done' checking. + - You can use it in all viewer-based scripts (random_policy, evaluate_ppo_from_checkpoint, etc.). + """ + + def __init__(self): + self.simulation_terminated = False + + def update_simulation_terminated(self, terminations, truncations): + """Update termination flag — should be called only after env.step().""" + self.simulation_terminated = terminations.get("__all__", False) or truncations.get("__all__", False) + + def should_step(self, control): + """ + Determine if env.step() should be performed in this loop iteration. + Returns True if: + - not paused + - or single step requested. + """ + return not control.paused or control.step_once