diff --git a/.DS_Store b/.DS_Store index 1b83a3bed..1a9d76103 100644 Binary files a/.DS_Store and b/.DS_Store differ diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..fbfd20763 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,94 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Build Commands + +```bash +# Build (skip tests) +mvn clean package -DskipTests + +# Run unit tests +mvn test + +# Run a single test class +mvn test -Dtest=PersonTest + +# Run all tests including integration tests +mvn verify +``` + +The build produces two runnable JARs: +- `target/singlerun.jar` — single simulation run (GUI or headless) +- `target/multirun.jar` — batch runs from a YAML config file + +## Running the Simulation + +```bash +# Single run (headless, UK, setup from scratch) +java -jar target/singlerun.jar -g false -c UK -Setup + +# Multi-run batch from config +java -jar target/multirun.jar -config config/default.yml -g false +``` + +Key CLI flags: `-c` (country), `-s` (start year), `-e` (end year), `-g` (GUI true/false), `-Setup` (rebuild database), `-r` (random seed), `-p` (population size). + +## Architecture + +SimPaths is a discrete-time (annual steps) agent-based microsimulation framework built on the [JAS-mine](https://www.jas-mine.net/) engine. It projects life histories forward across labour, family, health, and financial domains. + +### Agent Hierarchy + +``` +Household → BenefitUnit(s) → Person(s) +``` + +- **Person** (`simpaths/model/Person.java`) — individual agent; carries all demographics, health, education, labour, and income state. +- **BenefitUnit** (`simpaths/model/BenefitUnit.java`) — tax/benefit assessment unit (one or two adults + dependents). +- **Household** (`simpaths/model/Household.java`) — grouping of benefit units at the same address. + +### Package Map + +| Package | Responsibility | +|---|---| +| `simpaths/experiment/` | Entry points and orchestration: `SimPathsStart`, `SimPathsMultiRun`, `SimPathsCollector`, `SimPathsObserver` | +| `simpaths/model/` | Core simulation logic: agent classes, annual process methods, alignment, labour market, tax evaluation, intertemporal decisions | +| `simpaths/data/` | Parameters, setup routines, input parsers, filters, statistics helpers, regression managers, EUROMOD donor matching | + +### Simulation Engine + +`SimPathsModel.java` is the central manager registered with JAS-mine. It owns all agent collections and builds the ordered event schedule. Each simulated year runs **44 ordered processes** covering: +1. Year setup / parameter updates +2. Demographic events (ageing, mortality, fertility, education) +3. Labour market transitions +4. Partnership dynamics (cohabitation, separation, union matching via `UnionMatching.java`) +5. Health and wellbeing +6. Tax-benefit evaluation (via EUROMOD donor matching in `TaxEvaluation.java`) +7. Financial outcomes and aggregate alignment to calibration targets + +### Configuration System + +Runtime parameters live in `config/default.yml` (template) and are loaded by `SimPathsMultiRun`. The layered override order is: **class defaults → YAML values → CLI flags**. + +Key top-level YAML keys: `maxNumberOfRuns`, `executeWithGui`, `randomSeed`, `startYear`, `endYear`, `popSize`. Model-specific keys toggle alignment, time-trend controls, and individual module switches. + +### Data / Database + +The initial population and EUROMOD donor data are stored in an embedded **H2 database** built during the `-Setup` phase. Integration tests that rebuild or query the database are in `src/test/java/simpaths/integrationtest/`. + +## Key Tech + +- **Java 19**, Maven 3.x +- **JAS-mine 4.3.25** — microsimulation engine and GUI +- **JUnit 5 + Mockito 5** for tests +- **Apache Commons Math3, CLI, CSV** and **SnakeYAML** for utilities + +## Documentation + +Detailed guides are in `documentation/`: +- `model-concepts.md` — agent lifecycle and annual-cycle detail +- `configuration.md` — YAML structure, config keys, and how to write your own +- `data-pipeline.md` — how input data is prepared and loaded +- `validation-guide.md` — model validation procedures +- `cli-reference.md` — full CLI argument reference \ No newline at end of file diff --git a/config/default.yml b/config/default.yml index 631b016c3..ca449ce17 100644 --- a/config/default.yml +++ b/config/default.yml @@ -1,89 +1,177 @@ -# This file can be used to override defaults for multirun arguments. -# Arguments of the SimPathsMultiRun object overridden by the command-line - -maxNumberOfRuns: 1 -executeWithGui: false -randomSeed: 606 -startYear: 2019 -endYear: 2022 -popSize: 50000 -# countryString: "United Kingdom" -# integrationTest: false - -# Arguments passed to the SimPathsModel +# SimPaths multi-run configuration file. +# Uncomment and edit any field to override its default value. +# CLI flags take final precedence over anything set here. + +# ── Top-level run arguments ──────────────────────────────────────────────────── + +maxNumberOfRuns: 1 # number of sequential simulation runs +executeWithGui: false # true = launch JAS-mine GUI; false = headless (required on servers/CI) +randomSeed: 606 # seed for the first run; incremented automatically if randomSeedInnov is true +startYear: 2019 # first year of simulation (must have matching input/donor data) +endYear: 2022 # last year of simulation (inclusive) +popSize: 50000 # simulated population size (larger = more accurate, slower) +# countryString: "United Kingdom" # "United Kingdom" or "Italy" (auto-detected from donor DB if omitted) +# integrationTest: false # true = write output to a fixed folder for comparison in CI tests + + +# ── model_args: passed to SimPathsModel ─────────────────────────────────────── +# All keys map directly to @GUIparameter fields on SimPathsModel. +# Values shown are the class defaults. + model_args: -# maxAge: 130 -# fixTimeTrend: true -# timeTrendStopsIn: 2017 -# timeTrendStopsInMonetaryProcesses: 2017 -# fixRandomSeed: true -# sIndexTimeWindow: 5 -# sIndexAlpha: 2 -# sIndexDelta: 0 -# savingRate: 0 -# initialisePotentialEarningsFromDatabase: true -# useWeights: false -# useSBAMMatching: -# projectMortality: true -# alignPopulation: true -# alignFertility: true -# alignEducation: false -# alignInSchool: false -# alignCohabitation: false -# labourMarketCovid19On: false -# projectFormalChildcare: true -# donorPoolAveraging: true -# alignEmployment: false -# projectSocialCare: false -# addRegressionStochasticComponent: true -# fixRegressionStochasticComponent: false -# flagSuppressChildcareCosts: false -# flagSuppressSocialCareCosts: false + + # --- Time trend controls --- +# maxAge: 130 # maximum age kept in simulation; persons above this are removed +# fixTimeTrend: true # if true, freezes the time trend in regression equations +# timeTrendStopsIn: 2017 # year at which the time trend is frozen (if fixTimeTrend: true) +# timeTrendStopsInMonetaryProcesses: 2017 # same freeze year applied to monetary/income regressions only + + # --- Random number controls --- +# fixRandomSeed: true # if true, each run uses the same fixed seed (randomSeedIfFixed) + + # --- Income security (S-Index) --- + # The S-Index is an economic (in)security index computed from a rolling window of + # equivalised consumption, discounted and weighted by a risk-aversion parameter. + # SIndex_p50 is reported in Statistics1.csv each year. +# sIndexTimeWindow: 5 # length of rolling window in years (default 5) +# sIndexAlpha: 2 # coefficient of relative risk aversion (higher = more sensitivity to drops) +# sIndexDelta: 0.98 # annual discount factor applied to past consumption observations + + # --- Savings --- +# savingRate: 0.056 # fraction of equivalised disposable income saved (used when IO is disabled); + # default is OECD average UK household saving rate 2000–2019 + + # --- Wage initialisation --- +# initialisePotentialEarningsFromDatabase: true # initialise wage potential from donor DB rather than input CSV + + # --- Population weighting --- +# useWeights: false # if true, apply survey weights in alignment and statistics calculations + + # --- Matching method --- +# useSBAMMatching: # if true, use SBAM instead of standard union-matching algorithm + + # --- Demographic projections --- +# projectMortality: true # if false, disables stochastic mortality (population does not die) + + # --- Alignment flags --- + # See model-concepts.md for a full explanation of what alignment does. +# alignPopulation: true # align age-sex-region totals to official population projections +# alignFertility: true # scale birth probabilities to match projected fertility rates +# alignEducation: false # align completed education distribution to targets +# alignInSchool: false # align school participation rate (age 16–29) to targets +# alignCohabitation: false # align share of cohabiting individuals to targets +# alignEmployment: false # align employment share to targets + + # --- Labour market modules --- +# labourMarketCovid19On: false # enable reduced-form month-by-month COVID-19 labour market module + # (applies to years 2020–2021 in the baseline parameterisation) + + # --- Social care and childcare --- +# projectFormalChildcare: true # simulate formal childcare costs +# projectSocialCare: false # simulate social care receipt and provision module +# flagSuppressChildcareCosts: false # if true, set formal childcare costs to zero (scenario use) +# flagSuppressSocialCareCosts: false # if true, set social care costs to zero (scenario use) + + # --- Tax-benefit imputation --- +# donorPoolAveraging: true # if true, average disposable income over k nearest-neighbour donors + # rather than using the single closest donor; reduces imputation volatility + + # --- Regression stochasticity --- +# addRegressionStochasticComponent: true # include the residual draw in regression predictions +# fixRegressionStochasticComponent: false # if true, draw the residual once and hold it fixed + # across years (currently applies to wage equations only) + + # --- Time-series defaults --- +# flagDefaultToTimeSeriesAverages: false # if true, use the sample average of time-series variables + # rather than the year-specific value when data is unavailable + + # --- Intertemporal optimisation (IO) --- + # Enables backward-induction life-cycle solution for consumption and labour supply. + # Decision grids are pre-computed in year 0; agents look up optimal choices each year. + # Computationally intensive — disabled by default. # enableIntertemporalOptimisations: true -# flagDefaultToTimeSeriesAverages: true -# responsesToLowWageOffer: true -# responsesToPension: false -# saveImperfectTaxDBMatches: false -# useSavedBehaviour: false -# readGrid: "laptop serial" -# saveBehaviour: true -# employmentOptionsOfPrincipalWorker: 3 -# employmentOptionsOfSecondaryWorker: 3 -# responsesToEducation: true -# responsesToRetirement: false -# responsesToHealth: true -# responsesToDisability: false -# minAgeForPoorHealth: 50 -# responsesToRegion: false -# ignoreTargetsAtPopulationLoad: false - -# Arguments that alter processing of the SimPathsMultiRun object + + # IO state-space: which characteristics agents respond to when choosing labour/consumption. + # Each flag adds a dimension to the grid and increases solve time. +# responsesToHealth: true # include physical health in IO state space +# responsesToDisability: false # include disability status in IO state space +# responsesToEducation: true # include student and education level in IO state space +# responsesToPension: false # include private pension wealth in IO state space +# responsesToRetirement: false # include retirement state (and private pension) in IO state space +# responsesToLowWageOffer: true # include unemployment/low-wage-offer risk in IO state space +# responsesToRegion: false # include geographic region in IO state space +# minAgeForPoorHealth: 45 # minimum age from which less-than-perfect health enters state space + + # IO employment options +# employmentOptionsOfPrincipalWorker: 3 # number of discrete hours options for the principal earner +# employmentOptionsOfSecondaryWorker: 3 # number of discrete hours options for the secondary earner + + # IO grid persistence — save/reuse pre-computed grids across runs +# saveBehaviour: true # save decision grids to output folder after solving +# useSavedBehaviour: false # load grids from a previous run instead of recomputing +# readGrid: "test1" # name of the run whose grids to load (must match a folder in output/) + + # IO diagnostics +# saveImperfectTaxDBMatches: false # log cases where tax-benefit donor matching falls back to a coarser regime + + # --- Population load --- +# ignoreTargetsAtPopulationLoad: false # if true, skip alignment-target checks when loading the initial population + + +# ── innovation_args: parameter variation across sequential runs ──────────────── +# These flags control how parameters change between run 0, run 1, run 2, etc. +# Useful for sensitivity analysis and uncertainty quantification. + innovation_args: -# randomSeedInnov: false -# flagDatabaseSetup: false -# intertemporalElasticityInnov: false -# labourSupplyElasticityInnov: true +# randomSeedInnov: true # if true, increment randomSeed by 1 for each successive run + # (default true — each run gets a distinct seed) +# flagDatabaseSetup: false # if true, run database setup instead of simulation + # (equivalent to -DBSetup on the command line) +# intertemporalElasticityInnov: false # if true, applies interest rate shocks across runs: + # run 1: +0.0075 (higher return to saving) + # run 2: -0.0075 (lower return to saving) + # requires maxNumberOfRuns >= 3 to see all variants +# labourSupplyElasticityInnov: false # if true, applies disposable income shocks across runs: + # run 1: +0.01 (higher net labour income) + # run 2: -0.01 (lower net labour income) + # requires maxNumberOfRuns >= 3 to see all variants + + +# ── collector_args: output collection and export ─────────────────────────────── +# Controls what SimPathsCollector writes to CSV / database each year. +# +# Output files: +# Statistics1.csv — income distribution: Gini coefficients, income percentiles, median EDI, S-Index +# Statistics2.csv — demographic validation: partnership rates, employment, health, disability by age/gender +# Statistics3.csv — alignment diagnostics: simulated vs target rates and adjustment factors +# EmploymentStatistics.csv — labour market transitions and participation rates +# HealthStatistics.csv — health measures (SF-12, GHQ-12, EQ-5D) by age/gender collector_args: -# calculateGiniCoefficients: false -# exportToDatabase: false -# exportToCSV: true -# persistStatistics: true -# persistStatistics2: true -# persistStatistics3: true -# persistPersons: false -# persistBenefitUnits: false -# persistHouseholds: false -# persistEmploymentStatistics: false -# dataDumpStartTime: 0L -# dataDumpTimePeriod: 1.0 +# calculateGiniCoefficients: false # compute Gini coefficients (also populates GUI charts); off by default for speed +# exportToDatabase: false # write outputs to H2 database (in addition to or instead of CSV) +# exportToCSV: true # write outputs to CSV files under output//csv/ +# persistStatistics: true # write Statistics1.csv (income distribution) +# persistStatistics2: true # write Statistics2.csv (demographic validation outputs) +# persistStatistics3: true # write Statistics3.csv (alignment diagnostics) +# persistPersons: false # write one row per person per year (large files) +# persistBenefitUnits: false # write one row per benefit unit per year (large files) +# persistHouseholds: false # write one row per household per year +# persistEmploymentStatistics: false # write EmploymentStatistics.csv +# dataDumpStartTime: 0L # first year to write output (0 = startYear) +# dataDumpTimePeriod: 1.0 # output frequency in years (1.0 = every year) + + +# ── parameter_args: file paths and global flags ─────────────────────────────── parameter_args: -# input_directory: input -# input_directory_initial_populations: input/InitialPopulations -# euromod_output_directory: input/EUROMODoutput -# trainingFlag: false -# includeYears: +# input_directory: input # path to input data folder +# input_directory_initial_populations: input/InitialPopulations # path to initial population CSVs +# euromod_output_directory: input/EUROMODoutput # path to EUROMOD/UKMOD output files +# trainingFlag: false # if true, use training data from input/…/training/ subfolders + # (set automatically by test configs; do not set for research runs) +# includeYears: # list of policy years for which EUROMOD donor data is available; + # only these years will be included in the donor database # - 2011 # - 2012 # - 2013 @@ -96,4 +184,4 @@ parameter_args: # - 2020 # - 2021 # - 2022 -# - 2023 \ No newline at end of file +# - 2023 diff --git a/documentation/SimPaths Stata Parameters.xlsx b/documentation/SimPaths_Stata_Parameters.xlsx similarity index 100% rename from documentation/SimPaths Stata Parameters.xlsx rename to documentation/SimPaths_Stata_Parameters.xlsx diff --git a/documentation/repository-guide.md b/documentation/repository-guide.md new file mode 100644 index 000000000..e571662e2 --- /dev/null +++ b/documentation/repository-guide.md @@ -0,0 +1,533 @@ +# SimPaths Repository Guide + +A guide to navigating the SimPaths repository structure and codebase. + +--- + +## Table of Contents + +1. [Repository Structure](#repository-structure) +2. [Core Components](#core-components) +3. [Key Directories Explained](#key-directories-explained) +4. [Sub-package Detail](#sub-package-detail) +5. [Data Pipeline Reference](#data-pipeline-reference) +6. [Development Workflow](#development-workflow) +7. [Code Navigation Tips](#code-navigation-tips) +8. [Additional Resources](#additional-resources) + +--- + +## Repository Structure + +``` +SimPaths/ +├── config/ # Configuration files for simulations +│ ├── default.yml # Default simulation parameters +│ ├── test_create_database.yml # Database creation test config +│ └── test_run.yml # Test run configuration +│ +├── documentation/ # Comprehensive documentation +│ ├── figures/ # Diagrams and illustrations +│ ├── wiki/ # Full documentation website +│ │ ├── getting-started/ # Setup and first simulation guides +│ │ ├── overview/ # Model description and modules +│ │ ├── user-guide/ # Running simulations +│ │ ├── developer-guide/ # Extending the model +│ │ │ └── repository-guide.md # Repository guide (copy for website) +│ │ ├── jasmine-reference/ # JAS-mine library reference +│ │ ├── research/ # Published papers +│ │ └── validation/ # Model validation results +│ ├── repository-guide.md # Repository structure and navigation guide +│ ├── SimPaths_Variable_Codebook.xlsx # Codebook of all variables in SimPaths +│ ├── SimPaths_Stata_Parameters.xlsx # Comparison of parameters: Stata do-files vs Java code +│ └── SimPathsUK_Schedule.xlsx # Detailed schedule of events and corresponding classes +│ +├── input/ # Input data and parameters +│ ├── InitialPopulations/ # Starting population data +│ │ ├── training/ # De-identified training population (included in repo) +│ │ └── compile/ # Stata pipeline: builds populations, estimates regressions +│ │ ├── do_emphist/ # Employment history reconstruction sub-pipeline +│ │ └── RegressionEstimates/ # Regression coefficient estimation scripts +│ ├── EUROMODoutput/ # Tax-benefit model outputs +│ │ └── training/ # Training UKMOD outputs (included in repo) +│ ├── DoFilesTarget/ # Stata scripts that generate alignment targets +│ ├── align_*.xlsx # Alignment files (population, employment, etc.) +│ ├── reg_*.xlsx # Regression parameter files +│ ├── scenario_*.xlsx # Scenario configuration files +│ ├── projections_*.xlsx # Mortality/fertility projections +│ ├── DatabaseCountryYear.xlsx # Database metadata +│ ├── EUROMODpolicySchedule.xlsx # Policy schedule +│ ├── policy parameters.xlsx # Tax-benefit parameters +│ ├── validation_statistics.xlsx # Validation targets +│ └── input.mv.db # H2 donor database (generated by setup) +│ +├── output/ # Simulation outputs +│ ├── [timestamp]_[seed]_[run]/ # Timestamped output folders +│ │ ├── csv/ +│ │ │ ├── Statistics1.csv # Income distribution, Gini, S-Index +│ │ │ ├── Statistics2.csv # Demographics by age and gender +│ │ │ ├── Statistics3.csv # Alignment diagnostics +│ │ │ ├── Person.csv # Person-level output +│ │ │ ├── BenefitUnit.csv # Benefit-unit-level output +│ │ │ └── Household.csv # Household-level output +│ │ ├── database/ # Run-specific persistence output +│ │ └── input/ # Copied run input artifacts +│ └── logs/ # Log files (with -f flag on multirun) +│ +├── src/ # Source code +│ ├── main/ +│ │ ├── java/simpaths/ +│ │ │ ├── data/ # Data handling and parameters +│ │ │ ├── experiment/ # Simulation execution classes +│ │ │ └── model/ # Core model implementation +│ │ │ ├── decisions/ # Intertemporal optimisation grids +│ │ │ ├── enums/ # Categorical variable definitions +│ │ │ ├── taxes/ # EUROMOD donor matching +│ │ │ └── lifetime_incomes/ # Synthetic income trajectory generation +│ │ └── resources/ # Configuration resources +│ └── test/ # Test classes +│ +├── validation/ # Validation scripts and results +│ ├── 01_estimate_validation/ # Estimation validation +│ └── 02_simulated_output_validation/ # Output validation +│ +├── pom.xml # Maven build configuration +├── singlerun.jar # Executable for single runs +├── multirun.jar # Executable for multiple runs +└── README.md # Project overview +``` + +--- + +## Core Components + +### 1. **Entry Points** + +#### SimPathsStart (`src/main/java/simpaths/experiment/SimPathsStart.java`) +- Main class for single simulation execution +- Handles GUI and command-line interfaces +- Manages database setup phases +- **Key methods**: + - `main()`: Entry point + - `runGUIdialog()`: Launch GUI + - `runGUIlessSetup()`: Command-line setup + +#### SimPathsMultiRun (`src/main/java/simpaths/experiment/SimPathsMultiRun.java`) +- Coordinates multiple simulation runs +- Manages parallel execution +- Aggregates results across runs +- Configurable via YAML files + +### 2. **Core Model** + +#### SimPathsModel (`src/main/java/simpaths/model/SimPathsModel.java`) +- Central simulation manager +- Implements `AbstractSimulationManager` from JAS-mine +- Defines the simulation schedule via `buildSchedule()` +- Manages all simulation modules and processes +- **Key responsibilities**: + - Population initialization + - Event scheduling + - Module coordination + - Time progression + +### 3. **Data & Parameters** + +#### Parameters (`src/main/java/simpaths/data/Parameters.java`) +- Global parameter storage +- Loads regression coefficients from Excel +- Manages country-specific configurations +- Stores alignment targets +- **Key data structures**: + - Regression coefficient maps + - Policy parameters + - Alignment targets + - EUROMOD variable definitions + +--- + +## Key Directories Explained + +### `/src/main/java/simpaths/` + +#### `data/` +**Purpose**: Data handling, parameter management, and utility classes + +- **Parameters.java**: Global parameter storage and Excel data loading +- **ManagerRegressions.java**: Regression coefficient management +- **CallEUROMOD.java** / **CallEMLight.java**: Interface with tax-benefit models +- **filters/**: Collection filters for querying simulated populations +- **startingpop/**: Initial population data parsing +- **statistics/**: Statistical utilities + +#### `experiment/` +**Purpose**: Simulation execution and coordination + +- **SimPathsStart.java**: Single-run entry point +- **SimPathsMultiRun.java**: Multi-run orchestration +- **SimPathsCollector.java**: Output collection and aggregation +- **SimPathsObserver.java**: GUI updates and monitoring + +#### `model/` +**Purpose**: Core simulation logic + +- **SimPathsModel.java**: Main simulation manager +- **Person.java**: Individual-level processes and attributes +- **BenefitUnit.java**: Fiscal unit processes +- **Household.java**: Residential unit processes +- **decisions/**: Labour supply and consumption optimization +- **enums/**: Type-safe enumerations (Gender, Country, HealthStatus, etc.) +- **taxes/**: Tax-benefit donor matching system +- **lifetime_incomes/**: Lifetime income projection utilities + +### `/input/` + +**Critical input files**: + +| File Pattern | Purpose | +|--------------|---------| +| `align_*.xlsx` | Alignment targets (population, employment, education, etc.) | +| `reg_*.xlsx` | Regression parameters for behavioral processes | +| `scenario_*.xlsx` | Policy scenarios and projections | +| `projections_*.xlsx` | Demographic projections (mortality, fertility) | +| `DatabaseCountryYear.xlsx` | Tracks current database country/year | +| `EUROMODpolicySchedule.xlsx` | Tax-benefit policy schedule | +| `policy parameters.xlsx` | Detailed policy parameters | + +**Subdirectories**: +- `InitialPopulations/`: Starting population databases +- `EUROMODoutput/`: Tax-benefit donor population data +- `DoFilesTarget/`: Stata-generated alignment targets + +### `/config/` + +YAML configuration files override default parameters. The main file is **default.yml**, which contains several configuration sections: + +- **model_args**: SimPathsModel parameters (alignment switches, behavioral responses) +- **collector_args**: Output options (CSV, database, statistics) +- **parameter_args**: Data directories and input years +- **innovation_args**: Experimental parameters for sensitivity analysis + +Additional configuration files for testing: **test_create_database.yml**, **test_run.yml** + +### `/documentation/wiki/` + +Complete documentation organized by audience: + +- **getting-started/**: Environment setup, data access, first simulation +- **overview/**: Model description, modules, parameterization +- **user-guide/**: GUI, parameter modification, multiple runs +- **developer-guide/**: JAS-mine architecture, internals, how-to guides +- **jasmine-reference/**: Statistical packages, alignment, regression tools +- **research/**: Published papers and validation results + +--- + +## Sub-package Detail + +The following sub-packages are self-contained subsystems whose internals are not obvious from the class names alone. + +### `model/decisions/` — IO engine + +When IO is enabled, computing optimal consumption–labour choices for every agent at every time step would be prohibitively slow. This package solves the problem once before the simulation runs: it constructs a grid covering all meaningful combinations of state variables (wealth, age, health, family status, etc.), then works backwards from the end of life to find the optimal choice at each grid point (backward induction). During the simulation, agents simply look up their current state in the pre-computed grid. + +| Class | Purpose | +| --- | --- | +| `DecisionParams` | Defines the state-space dimensions and grid parameters for the optimisation problem. | +| `ManagerPopulateGrids` | Populates the state-space grid points and evaluates value functions by backward induction. | +| `ManagerSolveGrids` | Solves for optimal policy at each grid point. | +| `ManagerFileGrids` | Reads and writes pre-computed grids to disk, so they can be reused across runs. | +| `Grids` | Container for the set of solved decision grids. | +| `States` | Enumerates the state variables that define each grid point. | +| `Expectations` / `LocalExpectations` | Computes expected future values over stochastic transitions. | +| `CESUtility` | CES utility function used in the optimisation. | + +### `model/taxes/` — EUROMOD donor matching + +Imputes taxes and benefits onto simulated benefit units by matching them to pre-computed EUROMOD donor records. + +| Class | Purpose | +| --- | --- | +| `DonorTaxImputation` | Main entry point. Implements the three-step matching process: coarse-exact matching on characteristics, income proximity filtering, and candidate selection/averaging. | +| `KeyFunction` / `KeyFunction1`–`4` | Four progressively relaxed matching-key definitions. The system tries the tightest key first and falls back through wider keys if no donors are found. | +| `DonorKeys` | Builds composite matching keys from benefit-unit characteristics. | +| `DonorTaxUnit` / `DonorPerson` | Represent the pre-computed EUROMOD donor records loaded from the database. | +| `CandidateList` | Ranked list of donor matches for a given benefit unit, sorted by income proximity. | +| `Match` / `Matches` | Store the final selected donor(s) and their imputed tax-benefit values. | + +The `taxes/database/` sub-package handles loading donor data from the H2 database into memory (`TaxDonorDataParser`, `DatabaseExtension`, `MatchIndices`). + +### `model/lifetime_incomes/` — synthetic income trajectories + +When IO is enabled, this package creates projected income paths for birth cohorts using an AR(2) process anchored to age-gender geometric means, and matches simulated persons to donor income profiles. + +| Class | Purpose | +| --- | --- | +| `ManagerProjectLifetimeIncomes` | Generates the synthetic income trajectory database for all birth cohorts in the simulation horizon. | +| `LifetimeIncomeImputation` | Matches each simulated person to a donor income trajectory via binary search on the income CDF. | +| `AnnualIncome` | Implements the AR(2) income process with age-gender anchoring. | +| `BirthCohort` | Groups individuals by birth year for cohort-level income projection. | +| `Individual` | Entity carrying age dummies and log GDP per capita for income regression. | + +CSV filenames follow the pattern `.csv`. With a single run the suffix is `1`; with multiple runs each run produces its own numbered file. + +For a description of the variables in output CSV files, see `documentation/SimPaths_Variable_Codebook.xlsx`. For a description of each `reg_*`, `align_*`, and `scenario_*` input file, see [Model Parameterisation](wiki/overview/parameterisation.md) on the website. + +--- + +## Data Pipeline Reference + +This section explains how the simulation-ready input files in `input/` are generated from raw survey data, and what to do if you need to update or extend them. + +The pipeline has three independent parts: (1) initial populations, (2) regression coefficients, (3) alignment targets. Each can be re-run separately. + +### Data sources + +| Source | Description | Access | +|--------|-------------|--------| +| **UKHLS** (Understanding Society) | Main household panel survey; waves 1 to O (UKDA-6614-stata) | Requires EUL licence from UK Data Service | +| **BHPS** (British Household Panel Survey) | Historical predecessor to UKHLS; used for pre-2009 employment history | Bundled with UKHLS EUL | +| **WAS** (Wealth and Assets Survey) | Biennial survey of household wealth; waves 1 to 7 (UKDA-7215-stata) | Requires EUL licence from UK Data Service | +| **EUROMOD / UKMOD** | Tax-benefit microsimulation system | See [Tax-Benefit Donors (UK)](wiki/getting-started/data/tax-benefit-donors-uk.md) on the website | + +### Part 1 — Initial populations (`input/InitialPopulations/compile/`) + +**What it produces:** Annual CSV files `population_initial_UK_.csv` used as the starting population for each simulation run. + +**Master script:** `input/InitialPopulations/compile/00_master.do` + +The pipeline runs in numbered stages: + +| Script | What it does | +|--------|-------------| +| `01_prepare_UKHLS_pooled_data.do` | Pools and standardises UKHLS waves | +| `02_create_UKHLS_variables.do` | Constructs all required variables (demographics, labour, health, income, wealth flags) and applies simulation-consistency rules (retirement as absorbing state, education age bounds, work/hours consistency) | +| `02_01_checks.do` | Data quality checks | +| `03_social_care_received.do` | Social care receipt variables | +| `04_social_care_provided.do` | Informal care provision variables | +| `05_create_benefit_units.do` | Groups individuals into benefit units (tax units) following UK tax-benefit rules | +| `06_reweight_and_slice.do` | Reweighting and year-specific slicing | +| `07_was_wealth_data.do` | Prepares Wealth and Assets Survey data | +| `08_wealth_to_ukhls.do` | Merges WAS wealth into UKHLS records | +| `09_finalise_input_data.do` | Final cleaning and formatting | +| `10_check_yearly_data.do` | Per-year consistency checks | +| `99_training_data.do` | Produces the de-identified training population committed to `input/InitialPopulations/training/` | + +#### Employment history sub-pipeline (`compile/do_emphist/`) + +Reconstructs each respondent's monthly employment history from January 2007 onwards by combining UKHLS and BHPS interview records. The output variable `liwwh` (months employed since Jan 2007) feeds into the labour supply models. + +| Script | Purpose | +|--------|---------| +| `00_Master_emphist.do` | Master; sets parameters and calls sub-scripts | +| `01_Intdate.do` – `07_Empcal1a.do` | Sequential stages: interview dating, BHPS linkage, employment spell reconstruction, new-entrant identification | + +### Part 2 — Regression coefficients (`input/InitialPopulations/compile/RegressionEstimates/`) + +**What it produces:** The `reg_*.xlsx` coefficient tables read by `Parameters.java` at simulation startup. + +**Master script:** `input/InitialPopulations/compile/RegressionEstimates/master.do` + +> **Note:** Income and union-formation regressions depend on predicted wages, so `reg_wages.do` must complete before `reg_income.do` and `reg_partnership.do`. All other scripts can run in any order. + +**Required Stata packages:** `fre`, `tsspell`, `carryforward`, `outreg2`, `oparallel`, `gologit2`, `winsor`, `reghdfe`, `ftools`, `require` + +| Script | Module | Method | +|--------|--------|--------| +| `reg_wages.do` | Hourly wages | Heckman selection model (males and females separately) | +| `reg_income.do` | Non-labour income | Hurdle model (selection + amount); requires predicted wages | +| `reg_partnership.do` | Partnership formation/dissolution | Probit; requires predicted wages | +| `reg_education.do` | Education transitions | Generalised ordered logit | +| `reg_fertility.do` | Fertility | Probit | +| `reg_health.do` | Physical health (SF-12 PCS) | Linear regression | +| `reg_health_mental.do` | Mental health (GHQ-12, SF-12 MCS) | Linear regression | +| `reg_health_wellbeing.do` | Life satisfaction | Linear regression | +| `reg_home_ownership.do` | Homeownership transitions | Probit | +| `reg_retirement.do` | Retirement | Probit | +| `reg_leave_parental_home.do` | Leaving parental home | Probit | +| `reg_socialcare.do` | Social care receipt and provision | Probit / ordered logit | +| `reg_unemployment.do` | Unemployment transitions | Probit | +| `reg_financial_distress.do` | Financial distress | Probit | +| `programs.do` | Shared utility programs called by the estimation scripts | — | +| `variable_update.do` | Prepares and recodes variables before estimation | — | + +After running, output Excel files are placed in `input/` (overwriting the existing `reg_*.xlsx` files). + +### Part 3 — Alignment targets (`input/DoFilesTarget/`) + +**What it produces:** The `align_*.xlsx` and `*_targets.xlsx` files that the alignment modules use to rescale simulated rates. + +| Script | Output file | +|--------|------------| +| `01_employment_shares_initpopdata.do` | `input/employment_targets.xlsx` — employment shares by benefit-unit subgroup and year | +| `01_inSchool_targets_initpopdata.do` | `input/inSchool_targets.xlsx` — school participation rates by year | +| `03_calculate_partneredShare_initialPop_BUlogic.do` | `input/partnered_share_targets.xlsx` — partnership shares by year | +| `03_calculate_partnership_target.do` | Supplementary partnership targets | +| `02_person_risk_employment_stats.do` | `employment_risk_emp_stats.csv` — person-level at-risk diagnostics used for employment alignment group construction | + +Population projection targets (`align_popProjections.xlsx`) and fertility/mortality projections (`projections_*.xlsx`) come from ONS published projections and are not generated by these scripts. + +### When to re-run each part + +| Situation | What to re-run | +|-----------|---------------| +| Adding a new data year to the simulation | Part 1 (re-slice the population for the new year) + Part 3 (update alignment targets) | +| Re-estimating a behavioural module | Part 2 (the affected `reg_*.do` script only) + Stage 1 validation | +| Updating employment alignment targets | Part 3 (`01_employment_shares_initpopdata.do`) | + +After re-running any part, re-run setup (`singlerun -Setup` or `multirun -DBSetup`) to rebuild `input/input.mv.db` before running the simulation. + +### Setup-generated artifacts + +Running setup (`multirun -DBSetup`) creates or refreshes three files in `input/`: + +- `input.mv.db` — H2 database of EUROMOD donor tax-benefit outcomes +- `EUROMODpolicySchedule.xlsx` — maps simulation years to EUROMOD policy systems +- `DatabaseCountryYear.xlsx` — year-specific macro parameters + +These must exist before any simulation run. If they are missing, re-run setup. + +### Training mode + +The repository includes de-identified training data under `input/InitialPopulations/training/` and `input/EUROMODoutput/training/`. If no initial-population CSV files are found in the main input location, SimPaths automatically switches to training mode. Training mode supports development and CI but is not intended for research interpretation. + +### Logging + +With `-f` on `multirun.jar`, logs are written to `output/logs/run_.txt` (stdout) and `output/logs/run_.log` (log4j). + +--- + +## Development Workflow + +### 1. Understanding the Code + +**Start here**: +1. `SimPathsStart.java` — Understand initialization +2. `SimPathsModel.java` — Understand the simulation loop (`buildSchedule()`) +3. `Person.java`, `BenefitUnit.java`, `Household.java` — Understand agents +4. Module-specific methods in `Person.java` (e.g., `health()`, `education()`, `fertility()`) + +### 2. Key Design Patterns + +**JAS-mine Event Scheduling**: +```java +// In SimPathsModel.buildSchedule() +getEngine().getEventQueue().scheduleRepeat( + new SingleTargetEvent(this, Processes.UpdateYear), + 0.0, // Start time + 1.0 // Repeat interval +); +``` + +**Regression-based processes**: +```java +double score = Parameters.getRegression(RegressionName.HealthMentalHMLevel) + .getScore(regressors, Person.class.getDeclaredField("les_c4_lag1")); +``` + +**Alignment**: +```java +ResamplingAlignment.align( + population, // Collection to align + filter, // Subgroup filter + closure, // Alignment closure + targetValue // Target to match +); +``` + +### 3. Adding New Features + +**Example: Add a new person attribute** + +1. **Add field** to `Person.java`: + ```java + private Integer newAttribute; + ``` + +2. **Add getter/setter**: + ```java + public Integer getNewAttribute() { return newAttribute; } + public void setNewAttribute(Integer value) { this.newAttribute = value; } + ``` + +3. **Initialize** in constructor or relevant process method + +4. **Update database schema** if persisting (in `PERSON_VARIABLES_INITIAL`) + +5. **Add to outputs** in `SimPathsCollector.java` if needed + +**See**: `documentation/wiki/developer-guide/how-to/new-variable.md` + +### 4. Modifying Parameters + +**Regression coefficients**: Edit Excel files in `input/reg_*.xlsx` + +**Policy parameters**: Edit `input/policy parameters.xlsx` + +**Alignment targets**: Edit `input/align_*.xlsx` + +**Simulation options**: Edit `config/default.yml` or use GUI + +### 5. Adding GUI Parameters + +**Example**: +```java +@GUIparameter(description = "Enable new feature") +private Boolean enableNewFeature = true; +``` + +This automatically adds the parameter to the GUI interface. + +**See**: `documentation/wiki/developer-guide/how-to/add-gui-parameters.md` + +### 6. Testing + +Run tests via: +```bash +mvn test +``` + +Or via IDE test runner. + +### 7. Version Control + +**Branch naming conventions**: +- `feature/your-feature-name` — New features +- `bugfix/issue-number-description` — Bug fixes +- `docs/documentation-topic` — Documentation updates +- `experimental/your-description` — Experimental work + +**Main branches**: +- `main` — Stable release +- `develop` — Development integration + +--- + +## Code Navigation Tips + +**Find where a process runs**: +1. Search for the process name in `SimPathsModel.buildSchedule()` +2. Follow the method call to the implementation + +**Find regression parameters**: +1. Search for `Parameters.getRegression(RegressionName.XXX)` +2. The corresponding Excel file is in `input/reg_XXX.xlsx` + +**Find alignment logic**: +1. Search for classes ending in `Alignment` (e.g., `FertilityAlignment.java`) +2. Check `buildSchedule()` for when alignment occurs + +**Understand data flow**: +1. **Input**: Excel files → `Parameters.java` → Coefficient maps +2. **Process**: Regression score → Probability → Random draw → State change +3. **Output**: `SimPathsCollector.java` → CSV/Database + +--- + +## Additional Resources + +- **Full Documentation**: See `documentation/wiki/` for comprehensive guides +- **Getting Started**: `documentation/wiki/getting-started/` +- **Running Simulations**: `documentation/wiki/user-guide/` +- **Model Overview**: `documentation/wiki/overview/` +- **Issues**: [GitHub Issues](https://github.com/centreformicrosimulation/SimPaths/issues) diff --git a/documentation/wiki/assets/css/extra.css b/documentation/wiki/assets/css/extra.css index 04fef6eca..94ee99161 100644 --- a/documentation/wiki/assets/css/extra.css +++ b/documentation/wiki/assets/css/extra.css @@ -190,10 +190,6 @@ CONTENT — TYPOGRAPHY ═══════════════════════════════════════════════ */ -.md-content { - max-width: 820px; -} - .md-typeset { font-size: 0.82rem; line-height: 1.6; @@ -203,6 +199,7 @@ font-weight: 600; font-size: 1.45rem; letter-spacing: -0.015em; + color: var(--sp-primary); border-bottom: 2px solid transparent; border-image: var(--sp-gradient) 1; padding-bottom: 0.3rem; @@ -215,7 +212,7 @@ letter-spacing: -0.01em; margin-top: 1.4rem; margin-bottom: 0.45rem; - color: var(--md-default-fg-color); + color: var(--sp-primary); } .md-typeset h3 { @@ -223,7 +220,13 @@ font-size: 1.1rem; margin-top: 1rem; margin-bottom: 0.3rem; - color: var(--md-default-fg-color); + color: var(--sp-primary); +} + +[data-md-color-scheme="slate"] .md-typeset h1, +[data-md-color-scheme="slate"] .md-typeset h2, +[data-md-color-scheme="slate"] .md-typeset h3 { + color: #a8c8e8; } /* Paragraph justification */ diff --git a/documentation/wiki/developer-guide/internals/file-organisation.md b/documentation/wiki/developer-guide/internals/file-organisation.md deleted file mode 100644 index 742a1ef3a..000000000 --- a/documentation/wiki/developer-guide/internals/file-organisation.md +++ /dev/null @@ -1,5 +0,0 @@ -# File Organisation - -!!! warning "In progress" - This page is under development. Contributions welcome — - see the [Developer Guide](../index.md) for how to contribute. diff --git a/documentation/wiki/developer-guide/repository-guide.md b/documentation/wiki/developer-guide/repository-guide.md new file mode 100644 index 000000000..e571662e2 --- /dev/null +++ b/documentation/wiki/developer-guide/repository-guide.md @@ -0,0 +1,533 @@ +# SimPaths Repository Guide + +A guide to navigating the SimPaths repository structure and codebase. + +--- + +## Table of Contents + +1. [Repository Structure](#repository-structure) +2. [Core Components](#core-components) +3. [Key Directories Explained](#key-directories-explained) +4. [Sub-package Detail](#sub-package-detail) +5. [Data Pipeline Reference](#data-pipeline-reference) +6. [Development Workflow](#development-workflow) +7. [Code Navigation Tips](#code-navigation-tips) +8. [Additional Resources](#additional-resources) + +--- + +## Repository Structure + +``` +SimPaths/ +├── config/ # Configuration files for simulations +│ ├── default.yml # Default simulation parameters +│ ├── test_create_database.yml # Database creation test config +│ └── test_run.yml # Test run configuration +│ +├── documentation/ # Comprehensive documentation +│ ├── figures/ # Diagrams and illustrations +│ ├── wiki/ # Full documentation website +│ │ ├── getting-started/ # Setup and first simulation guides +│ │ ├── overview/ # Model description and modules +│ │ ├── user-guide/ # Running simulations +│ │ ├── developer-guide/ # Extending the model +│ │ │ └── repository-guide.md # Repository guide (copy for website) +│ │ ├── jasmine-reference/ # JAS-mine library reference +│ │ ├── research/ # Published papers +│ │ └── validation/ # Model validation results +│ ├── repository-guide.md # Repository structure and navigation guide +│ ├── SimPaths_Variable_Codebook.xlsx # Codebook of all variables in SimPaths +│ ├── SimPaths_Stata_Parameters.xlsx # Comparison of parameters: Stata do-files vs Java code +│ └── SimPathsUK_Schedule.xlsx # Detailed schedule of events and corresponding classes +│ +├── input/ # Input data and parameters +│ ├── InitialPopulations/ # Starting population data +│ │ ├── training/ # De-identified training population (included in repo) +│ │ └── compile/ # Stata pipeline: builds populations, estimates regressions +│ │ ├── do_emphist/ # Employment history reconstruction sub-pipeline +│ │ └── RegressionEstimates/ # Regression coefficient estimation scripts +│ ├── EUROMODoutput/ # Tax-benefit model outputs +│ │ └── training/ # Training UKMOD outputs (included in repo) +│ ├── DoFilesTarget/ # Stata scripts that generate alignment targets +│ ├── align_*.xlsx # Alignment files (population, employment, etc.) +│ ├── reg_*.xlsx # Regression parameter files +│ ├── scenario_*.xlsx # Scenario configuration files +│ ├── projections_*.xlsx # Mortality/fertility projections +│ ├── DatabaseCountryYear.xlsx # Database metadata +│ ├── EUROMODpolicySchedule.xlsx # Policy schedule +│ ├── policy parameters.xlsx # Tax-benefit parameters +│ ├── validation_statistics.xlsx # Validation targets +│ └── input.mv.db # H2 donor database (generated by setup) +│ +├── output/ # Simulation outputs +│ ├── [timestamp]_[seed]_[run]/ # Timestamped output folders +│ │ ├── csv/ +│ │ │ ├── Statistics1.csv # Income distribution, Gini, S-Index +│ │ │ ├── Statistics2.csv # Demographics by age and gender +│ │ │ ├── Statistics3.csv # Alignment diagnostics +│ │ │ ├── Person.csv # Person-level output +│ │ │ ├── BenefitUnit.csv # Benefit-unit-level output +│ │ │ └── Household.csv # Household-level output +│ │ ├── database/ # Run-specific persistence output +│ │ └── input/ # Copied run input artifacts +│ └── logs/ # Log files (with -f flag on multirun) +│ +├── src/ # Source code +│ ├── main/ +│ │ ├── java/simpaths/ +│ │ │ ├── data/ # Data handling and parameters +│ │ │ ├── experiment/ # Simulation execution classes +│ │ │ └── model/ # Core model implementation +│ │ │ ├── decisions/ # Intertemporal optimisation grids +│ │ │ ├── enums/ # Categorical variable definitions +│ │ │ ├── taxes/ # EUROMOD donor matching +│ │ │ └── lifetime_incomes/ # Synthetic income trajectory generation +│ │ └── resources/ # Configuration resources +│ └── test/ # Test classes +│ +├── validation/ # Validation scripts and results +│ ├── 01_estimate_validation/ # Estimation validation +│ └── 02_simulated_output_validation/ # Output validation +│ +├── pom.xml # Maven build configuration +├── singlerun.jar # Executable for single runs +├── multirun.jar # Executable for multiple runs +└── README.md # Project overview +``` + +--- + +## Core Components + +### 1. **Entry Points** + +#### SimPathsStart (`src/main/java/simpaths/experiment/SimPathsStart.java`) +- Main class for single simulation execution +- Handles GUI and command-line interfaces +- Manages database setup phases +- **Key methods**: + - `main()`: Entry point + - `runGUIdialog()`: Launch GUI + - `runGUIlessSetup()`: Command-line setup + +#### SimPathsMultiRun (`src/main/java/simpaths/experiment/SimPathsMultiRun.java`) +- Coordinates multiple simulation runs +- Manages parallel execution +- Aggregates results across runs +- Configurable via YAML files + +### 2. **Core Model** + +#### SimPathsModel (`src/main/java/simpaths/model/SimPathsModel.java`) +- Central simulation manager +- Implements `AbstractSimulationManager` from JAS-mine +- Defines the simulation schedule via `buildSchedule()` +- Manages all simulation modules and processes +- **Key responsibilities**: + - Population initialization + - Event scheduling + - Module coordination + - Time progression + +### 3. **Data & Parameters** + +#### Parameters (`src/main/java/simpaths/data/Parameters.java`) +- Global parameter storage +- Loads regression coefficients from Excel +- Manages country-specific configurations +- Stores alignment targets +- **Key data structures**: + - Regression coefficient maps + - Policy parameters + - Alignment targets + - EUROMOD variable definitions + +--- + +## Key Directories Explained + +### `/src/main/java/simpaths/` + +#### `data/` +**Purpose**: Data handling, parameter management, and utility classes + +- **Parameters.java**: Global parameter storage and Excel data loading +- **ManagerRegressions.java**: Regression coefficient management +- **CallEUROMOD.java** / **CallEMLight.java**: Interface with tax-benefit models +- **filters/**: Collection filters for querying simulated populations +- **startingpop/**: Initial population data parsing +- **statistics/**: Statistical utilities + +#### `experiment/` +**Purpose**: Simulation execution and coordination + +- **SimPathsStart.java**: Single-run entry point +- **SimPathsMultiRun.java**: Multi-run orchestration +- **SimPathsCollector.java**: Output collection and aggregation +- **SimPathsObserver.java**: GUI updates and monitoring + +#### `model/` +**Purpose**: Core simulation logic + +- **SimPathsModel.java**: Main simulation manager +- **Person.java**: Individual-level processes and attributes +- **BenefitUnit.java**: Fiscal unit processes +- **Household.java**: Residential unit processes +- **decisions/**: Labour supply and consumption optimization +- **enums/**: Type-safe enumerations (Gender, Country, HealthStatus, etc.) +- **taxes/**: Tax-benefit donor matching system +- **lifetime_incomes/**: Lifetime income projection utilities + +### `/input/` + +**Critical input files**: + +| File Pattern | Purpose | +|--------------|---------| +| `align_*.xlsx` | Alignment targets (population, employment, education, etc.) | +| `reg_*.xlsx` | Regression parameters for behavioral processes | +| `scenario_*.xlsx` | Policy scenarios and projections | +| `projections_*.xlsx` | Demographic projections (mortality, fertility) | +| `DatabaseCountryYear.xlsx` | Tracks current database country/year | +| `EUROMODpolicySchedule.xlsx` | Tax-benefit policy schedule | +| `policy parameters.xlsx` | Detailed policy parameters | + +**Subdirectories**: +- `InitialPopulations/`: Starting population databases +- `EUROMODoutput/`: Tax-benefit donor population data +- `DoFilesTarget/`: Stata-generated alignment targets + +### `/config/` + +YAML configuration files override default parameters. The main file is **default.yml**, which contains several configuration sections: + +- **model_args**: SimPathsModel parameters (alignment switches, behavioral responses) +- **collector_args**: Output options (CSV, database, statistics) +- **parameter_args**: Data directories and input years +- **innovation_args**: Experimental parameters for sensitivity analysis + +Additional configuration files for testing: **test_create_database.yml**, **test_run.yml** + +### `/documentation/wiki/` + +Complete documentation organized by audience: + +- **getting-started/**: Environment setup, data access, first simulation +- **overview/**: Model description, modules, parameterization +- **user-guide/**: GUI, parameter modification, multiple runs +- **developer-guide/**: JAS-mine architecture, internals, how-to guides +- **jasmine-reference/**: Statistical packages, alignment, regression tools +- **research/**: Published papers and validation results + +--- + +## Sub-package Detail + +The following sub-packages are self-contained subsystems whose internals are not obvious from the class names alone. + +### `model/decisions/` — IO engine + +When IO is enabled, computing optimal consumption–labour choices for every agent at every time step would be prohibitively slow. This package solves the problem once before the simulation runs: it constructs a grid covering all meaningful combinations of state variables (wealth, age, health, family status, etc.), then works backwards from the end of life to find the optimal choice at each grid point (backward induction). During the simulation, agents simply look up their current state in the pre-computed grid. + +| Class | Purpose | +| --- | --- | +| `DecisionParams` | Defines the state-space dimensions and grid parameters for the optimisation problem. | +| `ManagerPopulateGrids` | Populates the state-space grid points and evaluates value functions by backward induction. | +| `ManagerSolveGrids` | Solves for optimal policy at each grid point. | +| `ManagerFileGrids` | Reads and writes pre-computed grids to disk, so they can be reused across runs. | +| `Grids` | Container for the set of solved decision grids. | +| `States` | Enumerates the state variables that define each grid point. | +| `Expectations` / `LocalExpectations` | Computes expected future values over stochastic transitions. | +| `CESUtility` | CES utility function used in the optimisation. | + +### `model/taxes/` — EUROMOD donor matching + +Imputes taxes and benefits onto simulated benefit units by matching them to pre-computed EUROMOD donor records. + +| Class | Purpose | +| --- | --- | +| `DonorTaxImputation` | Main entry point. Implements the three-step matching process: coarse-exact matching on characteristics, income proximity filtering, and candidate selection/averaging. | +| `KeyFunction` / `KeyFunction1`–`4` | Four progressively relaxed matching-key definitions. The system tries the tightest key first and falls back through wider keys if no donors are found. | +| `DonorKeys` | Builds composite matching keys from benefit-unit characteristics. | +| `DonorTaxUnit` / `DonorPerson` | Represent the pre-computed EUROMOD donor records loaded from the database. | +| `CandidateList` | Ranked list of donor matches for a given benefit unit, sorted by income proximity. | +| `Match` / `Matches` | Store the final selected donor(s) and their imputed tax-benefit values. | + +The `taxes/database/` sub-package handles loading donor data from the H2 database into memory (`TaxDonorDataParser`, `DatabaseExtension`, `MatchIndices`). + +### `model/lifetime_incomes/` — synthetic income trajectories + +When IO is enabled, this package creates projected income paths for birth cohorts using an AR(2) process anchored to age-gender geometric means, and matches simulated persons to donor income profiles. + +| Class | Purpose | +| --- | --- | +| `ManagerProjectLifetimeIncomes` | Generates the synthetic income trajectory database for all birth cohorts in the simulation horizon. | +| `LifetimeIncomeImputation` | Matches each simulated person to a donor income trajectory via binary search on the income CDF. | +| `AnnualIncome` | Implements the AR(2) income process with age-gender anchoring. | +| `BirthCohort` | Groups individuals by birth year for cohort-level income projection. | +| `Individual` | Entity carrying age dummies and log GDP per capita for income regression. | + +CSV filenames follow the pattern `.csv`. With a single run the suffix is `1`; with multiple runs each run produces its own numbered file. + +For a description of the variables in output CSV files, see `documentation/SimPaths_Variable_Codebook.xlsx`. For a description of each `reg_*`, `align_*`, and `scenario_*` input file, see [Model Parameterisation](wiki/overview/parameterisation.md) on the website. + +--- + +## Data Pipeline Reference + +This section explains how the simulation-ready input files in `input/` are generated from raw survey data, and what to do if you need to update or extend them. + +The pipeline has three independent parts: (1) initial populations, (2) regression coefficients, (3) alignment targets. Each can be re-run separately. + +### Data sources + +| Source | Description | Access | +|--------|-------------|--------| +| **UKHLS** (Understanding Society) | Main household panel survey; waves 1 to O (UKDA-6614-stata) | Requires EUL licence from UK Data Service | +| **BHPS** (British Household Panel Survey) | Historical predecessor to UKHLS; used for pre-2009 employment history | Bundled with UKHLS EUL | +| **WAS** (Wealth and Assets Survey) | Biennial survey of household wealth; waves 1 to 7 (UKDA-7215-stata) | Requires EUL licence from UK Data Service | +| **EUROMOD / UKMOD** | Tax-benefit microsimulation system | See [Tax-Benefit Donors (UK)](wiki/getting-started/data/tax-benefit-donors-uk.md) on the website | + +### Part 1 — Initial populations (`input/InitialPopulations/compile/`) + +**What it produces:** Annual CSV files `population_initial_UK_.csv` used as the starting population for each simulation run. + +**Master script:** `input/InitialPopulations/compile/00_master.do` + +The pipeline runs in numbered stages: + +| Script | What it does | +|--------|-------------| +| `01_prepare_UKHLS_pooled_data.do` | Pools and standardises UKHLS waves | +| `02_create_UKHLS_variables.do` | Constructs all required variables (demographics, labour, health, income, wealth flags) and applies simulation-consistency rules (retirement as absorbing state, education age bounds, work/hours consistency) | +| `02_01_checks.do` | Data quality checks | +| `03_social_care_received.do` | Social care receipt variables | +| `04_social_care_provided.do` | Informal care provision variables | +| `05_create_benefit_units.do` | Groups individuals into benefit units (tax units) following UK tax-benefit rules | +| `06_reweight_and_slice.do` | Reweighting and year-specific slicing | +| `07_was_wealth_data.do` | Prepares Wealth and Assets Survey data | +| `08_wealth_to_ukhls.do` | Merges WAS wealth into UKHLS records | +| `09_finalise_input_data.do` | Final cleaning and formatting | +| `10_check_yearly_data.do` | Per-year consistency checks | +| `99_training_data.do` | Produces the de-identified training population committed to `input/InitialPopulations/training/` | + +#### Employment history sub-pipeline (`compile/do_emphist/`) + +Reconstructs each respondent's monthly employment history from January 2007 onwards by combining UKHLS and BHPS interview records. The output variable `liwwh` (months employed since Jan 2007) feeds into the labour supply models. + +| Script | Purpose | +|--------|---------| +| `00_Master_emphist.do` | Master; sets parameters and calls sub-scripts | +| `01_Intdate.do` – `07_Empcal1a.do` | Sequential stages: interview dating, BHPS linkage, employment spell reconstruction, new-entrant identification | + +### Part 2 — Regression coefficients (`input/InitialPopulations/compile/RegressionEstimates/`) + +**What it produces:** The `reg_*.xlsx` coefficient tables read by `Parameters.java` at simulation startup. + +**Master script:** `input/InitialPopulations/compile/RegressionEstimates/master.do` + +> **Note:** Income and union-formation regressions depend on predicted wages, so `reg_wages.do` must complete before `reg_income.do` and `reg_partnership.do`. All other scripts can run in any order. + +**Required Stata packages:** `fre`, `tsspell`, `carryforward`, `outreg2`, `oparallel`, `gologit2`, `winsor`, `reghdfe`, `ftools`, `require` + +| Script | Module | Method | +|--------|--------|--------| +| `reg_wages.do` | Hourly wages | Heckman selection model (males and females separately) | +| `reg_income.do` | Non-labour income | Hurdle model (selection + amount); requires predicted wages | +| `reg_partnership.do` | Partnership formation/dissolution | Probit; requires predicted wages | +| `reg_education.do` | Education transitions | Generalised ordered logit | +| `reg_fertility.do` | Fertility | Probit | +| `reg_health.do` | Physical health (SF-12 PCS) | Linear regression | +| `reg_health_mental.do` | Mental health (GHQ-12, SF-12 MCS) | Linear regression | +| `reg_health_wellbeing.do` | Life satisfaction | Linear regression | +| `reg_home_ownership.do` | Homeownership transitions | Probit | +| `reg_retirement.do` | Retirement | Probit | +| `reg_leave_parental_home.do` | Leaving parental home | Probit | +| `reg_socialcare.do` | Social care receipt and provision | Probit / ordered logit | +| `reg_unemployment.do` | Unemployment transitions | Probit | +| `reg_financial_distress.do` | Financial distress | Probit | +| `programs.do` | Shared utility programs called by the estimation scripts | — | +| `variable_update.do` | Prepares and recodes variables before estimation | — | + +After running, output Excel files are placed in `input/` (overwriting the existing `reg_*.xlsx` files). + +### Part 3 — Alignment targets (`input/DoFilesTarget/`) + +**What it produces:** The `align_*.xlsx` and `*_targets.xlsx` files that the alignment modules use to rescale simulated rates. + +| Script | Output file | +|--------|------------| +| `01_employment_shares_initpopdata.do` | `input/employment_targets.xlsx` — employment shares by benefit-unit subgroup and year | +| `01_inSchool_targets_initpopdata.do` | `input/inSchool_targets.xlsx` — school participation rates by year | +| `03_calculate_partneredShare_initialPop_BUlogic.do` | `input/partnered_share_targets.xlsx` — partnership shares by year | +| `03_calculate_partnership_target.do` | Supplementary partnership targets | +| `02_person_risk_employment_stats.do` | `employment_risk_emp_stats.csv` — person-level at-risk diagnostics used for employment alignment group construction | + +Population projection targets (`align_popProjections.xlsx`) and fertility/mortality projections (`projections_*.xlsx`) come from ONS published projections and are not generated by these scripts. + +### When to re-run each part + +| Situation | What to re-run | +|-----------|---------------| +| Adding a new data year to the simulation | Part 1 (re-slice the population for the new year) + Part 3 (update alignment targets) | +| Re-estimating a behavioural module | Part 2 (the affected `reg_*.do` script only) + Stage 1 validation | +| Updating employment alignment targets | Part 3 (`01_employment_shares_initpopdata.do`) | + +After re-running any part, re-run setup (`singlerun -Setup` or `multirun -DBSetup`) to rebuild `input/input.mv.db` before running the simulation. + +### Setup-generated artifacts + +Running setup (`multirun -DBSetup`) creates or refreshes three files in `input/`: + +- `input.mv.db` — H2 database of EUROMOD donor tax-benefit outcomes +- `EUROMODpolicySchedule.xlsx` — maps simulation years to EUROMOD policy systems +- `DatabaseCountryYear.xlsx` — year-specific macro parameters + +These must exist before any simulation run. If they are missing, re-run setup. + +### Training mode + +The repository includes de-identified training data under `input/InitialPopulations/training/` and `input/EUROMODoutput/training/`. If no initial-population CSV files are found in the main input location, SimPaths automatically switches to training mode. Training mode supports development and CI but is not intended for research interpretation. + +### Logging + +With `-f` on `multirun.jar`, logs are written to `output/logs/run_.txt` (stdout) and `output/logs/run_.log` (log4j). + +--- + +## Development Workflow + +### 1. Understanding the Code + +**Start here**: +1. `SimPathsStart.java` — Understand initialization +2. `SimPathsModel.java` — Understand the simulation loop (`buildSchedule()`) +3. `Person.java`, `BenefitUnit.java`, `Household.java` — Understand agents +4. Module-specific methods in `Person.java` (e.g., `health()`, `education()`, `fertility()`) + +### 2. Key Design Patterns + +**JAS-mine Event Scheduling**: +```java +// In SimPathsModel.buildSchedule() +getEngine().getEventQueue().scheduleRepeat( + new SingleTargetEvent(this, Processes.UpdateYear), + 0.0, // Start time + 1.0 // Repeat interval +); +``` + +**Regression-based processes**: +```java +double score = Parameters.getRegression(RegressionName.HealthMentalHMLevel) + .getScore(regressors, Person.class.getDeclaredField("les_c4_lag1")); +``` + +**Alignment**: +```java +ResamplingAlignment.align( + population, // Collection to align + filter, // Subgroup filter + closure, // Alignment closure + targetValue // Target to match +); +``` + +### 3. Adding New Features + +**Example: Add a new person attribute** + +1. **Add field** to `Person.java`: + ```java + private Integer newAttribute; + ``` + +2. **Add getter/setter**: + ```java + public Integer getNewAttribute() { return newAttribute; } + public void setNewAttribute(Integer value) { this.newAttribute = value; } + ``` + +3. **Initialize** in constructor or relevant process method + +4. **Update database schema** if persisting (in `PERSON_VARIABLES_INITIAL`) + +5. **Add to outputs** in `SimPathsCollector.java` if needed + +**See**: `documentation/wiki/developer-guide/how-to/new-variable.md` + +### 4. Modifying Parameters + +**Regression coefficients**: Edit Excel files in `input/reg_*.xlsx` + +**Policy parameters**: Edit `input/policy parameters.xlsx` + +**Alignment targets**: Edit `input/align_*.xlsx` + +**Simulation options**: Edit `config/default.yml` or use GUI + +### 5. Adding GUI Parameters + +**Example**: +```java +@GUIparameter(description = "Enable new feature") +private Boolean enableNewFeature = true; +``` + +This automatically adds the parameter to the GUI interface. + +**See**: `documentation/wiki/developer-guide/how-to/add-gui-parameters.md` + +### 6. Testing + +Run tests via: +```bash +mvn test +``` + +Or via IDE test runner. + +### 7. Version Control + +**Branch naming conventions**: +- `feature/your-feature-name` — New features +- `bugfix/issue-number-description` — Bug fixes +- `docs/documentation-topic` — Documentation updates +- `experimental/your-description` — Experimental work + +**Main branches**: +- `main` — Stable release +- `develop` — Development integration + +--- + +## Code Navigation Tips + +**Find where a process runs**: +1. Search for the process name in `SimPathsModel.buildSchedule()` +2. Follow the method call to the implementation + +**Find regression parameters**: +1. Search for `Parameters.getRegression(RegressionName.XXX)` +2. The corresponding Excel file is in `input/reg_XXX.xlsx` + +**Find alignment logic**: +1. Search for classes ending in `Alignment` (e.g., `FertilityAlignment.java`) +2. Check `buildSchedule()` for when alignment occurs + +**Understand data flow**: +1. **Input**: Excel files → `Parameters.java` → Coefficient maps +2. **Process**: Regression score → Probability → Random draw → State change +3. **Output**: `SimPathsCollector.java` → CSV/Database + +--- + +## Additional Resources + +- **Full Documentation**: See `documentation/wiki/` for comprehensive guides +- **Getting Started**: `documentation/wiki/getting-started/` +- **Running Simulations**: `documentation/wiki/user-guide/` +- **Model Overview**: `documentation/wiki/overview/` +- **Issues**: [GitHub Issues](https://github.com/centreformicrosimulation/SimPaths/issues) diff --git a/documentation/wiki/figures/Chart Properties.png b/documentation/wiki/figures/Chart Properties.png deleted file mode 100644 index 757653b1e..000000000 Binary files a/documentation/wiki/figures/Chart Properties.png and /dev/null differ diff --git a/documentation/wiki/figures/Charts.png b/documentation/wiki/figures/Charts.png deleted file mode 100644 index 1791402d8..000000000 Binary files a/documentation/wiki/figures/Charts.png and /dev/null differ diff --git a/documentation/wiki/figures/Output stream.png b/documentation/wiki/figures/Output stream.png deleted file mode 100644 index 01bdfa14f..000000000 Binary files a/documentation/wiki/figures/Output stream.png and /dev/null differ diff --git a/documentation/wiki/figures/SimPaths GUI.png b/documentation/wiki/figures/SimPaths GUI.png deleted file mode 100644 index 369771f9b..000000000 Binary files a/documentation/wiki/figures/SimPaths GUI.png and /dev/null differ diff --git a/documentation/wiki/figures/SimPaths parameters.png b/documentation/wiki/figures/SimPaths parameters.png deleted file mode 100644 index 96a4eab5a..000000000 Binary files a/documentation/wiki/figures/SimPaths parameters.png and /dev/null differ diff --git a/documentation/wiki/figures/SimPaths-Buttons.png b/documentation/wiki/figures/SimPaths-Buttons.png deleted file mode 100644 index 65dd272c5..000000000 Binary files a/documentation/wiki/figures/SimPaths-Buttons.png and /dev/null differ diff --git a/documentation/wiki/figures/SimPaths-Chart-Zoom.png b/documentation/wiki/figures/SimPaths-Chart-Zoom.png deleted file mode 100644 index 05bec29c0..000000000 Binary files a/documentation/wiki/figures/SimPaths-Chart-Zoom.png and /dev/null differ diff --git a/documentation/wiki/getting-started/environment-setup.md b/documentation/wiki/getting-started/environment-setup.md index d121cab7c..e3731590a 100644 --- a/documentation/wiki/getting-started/environment-setup.md +++ b/documentation/wiki/getting-started/environment-setup.md @@ -6,8 +6,8 @@ ## Requirements -- Java Development Kit (JDK) 11 or later -- Apache Maven 3.6 or later +- Java Development Kit (JDK) 19 (the project targets Java 19 — earlier versions will not compile) +- Apache Maven 3.8 or later - Git ## Cloning the repository @@ -20,7 +20,9 @@ cd SimPaths ## Building the project ```bash -mvn clean install -DskipTests +mvn clean package ``` +This produces `singlerun.jar` and `multirun.jar` at the repository root. + Refer to the [Working in GitHub](../developer-guide/working-in-github.md) guide for the full development workflow.