Refactor/monitoring single core

[acavelan]

Monitoring refactor

Summary

This PR replaces the fragmented monitoring subsystem with a smaller implementation that is easier to maintain, trace, and debug. It removes 858 net lines across 55 files.

A measure now has one identity throughout the model:

<option name="nHost"> -> mon::measure("nHost") -> one defs[] entry -> its stable output number

The main changes are:

• Consolidate the monitoring implementation and remove overlapping legacy layers.
• Use the exact XML measure name throughout the C++ code.
• Reduce the recording API to three functions with explicit survey semantics.
• Add an optional versioned binary output format with optional gzip compression.
• Simplify monitoring storage, checkpointing, and continuous reporting.

Consolidated architecture

Seven legacy monitoring files and their overlapping responsibilities were replaced by:

• Monitoring.{h,cpp} for runtime state, recording, checkpointing, and output.
• OutMeasures.h for measure definitions.
• init.{h,cpp} for scenario-driven initialization.

This removes the former AgeGroup, OutputMeasures, info, management, reporting, mon.cpp, and misc.cpp layers.

The old measure enums, such as MHR_HOSTS and MHF_LOG_DENSITY, were removed. Call sites now use the exact XML name:

mon::recordStat(mon::measure("nHost"), human);

Measure is a compact index into one constexpr defs[] table. That table is the source of truth for the measure name, stable output number, numeric formatting, and supported dimensions. C++ measure literals are resolved with consteval, so an unknown name fails at compile time.

The internalName proxy, measureKey() indirection, and deployment-method metadata were removed from measure definitions. Deployment variants are recorded directly under their final measure names.

Five category booleans were replaced by a Dim bitmask for age, cohort, species, genotype, and drug. Each configured output uses one flattened buffer selected from std::vector<int> or std::vector<double>, preserving compact integer storage without duplicating routing or output code.

Recording and survey semantics

Fifteen type- and dimension-specific reporting functions were replaced by three semantic names:

• recordStat() records values for the current reported survey.
• recordEvent() accumulates events toward the next reported survey.
• recordDeploy() selects the final deployment measure.

All dimensional routing and accumulation pass through one private recordValue() implementation, and all in-tree call sites were migrated.

The survey timeline retains both reported and unreported dates because both affect event attribution. Runtime state separately tracks the current survey and the next reported survey receiving events.

This also fixes nCMDTReport: custom decision-tree reports now use event semantics and are no longer lost when an unreported survey boundary occurs before the next report.

Text and binary output

Text and binary output use the same logical row generation. The new --output-format txt|bin option selects exactly one format:

• Text remains the default and preserves the existing format.
• Plain text is opened in binary mode so line endings remain LF on Windows.
• Either format may use --compress-output.
• Output names without an extension automatically receive .txt, .txt.gz, .bin, or .bin.gz according to the selected options.
• Explicit extensions must match the selected format and compression; inconsistent names are rejected during command-line parsing.

Binary output uses a versioned, little-endian columnar layout. Its header stores separate integer and floating-point row counts, followed by contiguous int32_t survey, category, and measure columns. Integer measure values are written as int32_t; floating-point measure values remain IEEE-754 doubles. Dimensionless measures use category 0, as in text output.

Grouping identifiers and values by column improves both direct reads and gzip compression. It also preserves the compact integer representation from monitoring buffers through serialization instead of converting every value to a double.

Binary format

The header is 28 bytes. Let I be the number of integer rows, D the number of floating-point rows, and N = I + D. All fields are little-endian, integer rows precede floating-point rows, and .bin.gz is the same payload wrapped in gzip.

Block	Representation	Entries
Magic	8 bytes: OMOUTB2\0	1
Version	uint32 (2)	1
Integer row count	uint64	1
Floating-point row count	uint64	1
Survey column	int32	N
Category (ageGroup) column	int32	N
Measure column	int32	N
Integer value column	int32	I
Floating-point value column	IEEE-754 double	D

Python reader

This reader requires NumPy and pandas. Plain files are memory-mapped; gzip files are decompressed once in memory.

import gzip
import struct

import numpy as np
import pandas as pd


def read_monitoring_binary(path):
    if str(path).endswith(".gz"):
        with gzip.open(path, "rb") as stream:
            data = stream.read()
    else:
        data = np.memmap(path, mode="r", dtype="u1")

    if len(data) < 28:
        raise ValueError(f"truncated binary output: {path}")

    magic, version, n_int, n_double = struct.unpack_from("<8sIQQ", data)
    if magic != b"OMOUTB2\0" or version != 2:
        raise ValueError(f"unsupported OpenMalaria binary format: {path}")

    n = n_int + n_double
    if len(data) != 28 + 12 * n + 4 * n_int + 8 * n_double:
        raise ValueError(f"invalid binary output size: {path}")

    ids = np.frombuffer(data, "<i4", 3 * n, 28).reshape(3, n)
    offset = 28 + 12 * n

    value = np.empty(n, dtype="f8")
    value[:n_int] = np.frombuffer(data, "<i4", n_int, offset)
    offset += 4 * n_int
    value[n_int:] = np.frombuffer(data, "<f8", n_double, offset)

    columns = dict(zip(("survey", "ageGroup", "measure"), ids))
    columns["value"] = value
    return pd.DataFrame(columns, copy=False)

R reader

This reader uses data.table, bulk readBin() calls, and streamed gzip input.

read_uint64 <- function(bytes) {
  sum(as.double(bytes) * 256^(0:7))
}

read_monitoring_binary <- function(path) {
  con <- if (endsWith(path, ".gz")) gzfile(path, "rb") else file(path, "rb")
  on.exit(close(con))

  header <- readBin(con, raw(), 28L)
  if (length(header) != 28L) stop("Truncated binary output: ", path)

  magic <- header[1:8]
  version <- readBin(header[9:12], integer(), 1L, 4L, endian = "little")
  integer_rows <- read_uint64(header[13:20])
  double_rows <- read_uint64(header[21:28])

  if (!identical(magic, c(charToRaw("OMOUTB2"), as.raw(0))) || version != 2L) {
    stop("Unsupported OpenMalaria binary format: ", path)
  }

  rows <- integer_rows + double_rows
  result <- list(
    survey = readBin(con, integer(), rows, 4L, endian = "little"),
    ageGroup = readBin(con, integer(), rows, 4L, endian = "little"),
    measure = readBin(con, integer(), rows, 4L, endian = "little"),
    value = c(
      readBin(con, integer(), integer_rows, 4L, endian = "little"),
      readBin(con, double(), double_rows, 8L, endian = "little")
    )
  )

  data.table::setDT(result)
  if (nrow(result) != rows || length(readBin(con, raw(), 1L))) {
    stop("Invalid binary output size: ", path)
  }
  result
}

Output files are checked after opening and explicit close. Invalid paths and delayed write failures such as a full disk now terminate with the existing FileIO status instead of returning success.

Checkpointing and continuous output

Monitoring report buffers are allocated from scenario-derived dimensions before checkpoint loading. Serialized lengths must exactly match those allocations, so monitoring does not need an arbitrary container limit and malformed checkpoints cannot control allocation size.

The empty ContinuousType singleton was replaced by mon::Continuous namespace functions. Fresh and checkpoint initialization now share option resolution, and continuous checkpointing stores the absolute output position directly.

The monitoring checkpoint layout changed, so cross-version checkpoint compatibility should not be assumed.

Compatibility

• No scenario schema changes.
• Existing XML measure names and output numbers are preserved.
• Existing text output remains the default.
• Custom output numbers must be non-negative; negative values now produce a scenario error instead of silently disappearing.
• Internal C++ monitoring APIs changed; all in-tree callers were migrated.
• Binary survey output is new, but optional.

Validation

• Assertion-enabled build and focused checkpoint, Genotypes, Vivax, and DecisionTree tests.
• Release build and all 52 unit/scenario tests.
• Plain and gzip binary output decode identically, and the decoded rows match the corresponding text output.

R 4.6 warm-cache medians for the complete extraction of 189,800 rows from ModelNameNoOverrides (15 batches of 20 reads, data.table 1.18.4 using four threads):

Format	Size	Median read
Text with fread()	2,054 KiB	3.75 ms
Gzip text with fread()	455 KiB	22.75 ms
Columnar binary with readBin()	3,068 KiB	1.95 ms
Gzip columnar binary streamed with gzfile() and readBin()	56 KiB	3.60 ms

The R readers preallocate the per-scenario result list and add the scenario index by reference with set(). The binary reader uses one readBin() call per contiguous column and converts the resulting named list to a data.table by reference with setDT(). Gzip input is streamed through gzfile() without an additional uncompressed-file-sized raw buffer. Plain binary was about twice as fast as plain text in this test; gzip binary was much smaller and substantially faster to read than gzip text.

Monitoring Refactor Changes

Continuous Outputs

• Replaced the empty ContinuousType class and global singleton with mon::Continuous namespace functions: init(), update(), checkpoint(), and registerCallback(); all initialization, simulation, and checkpoint callers now use these functions directly.
• Preserved all three callback overloads, the callback registry, selected-output list, XML option names, headings, reporting cadence, period, duringInit, and generated values.
• Fresh and checkpoint initialization now use the same option-selection and validation loop.
• Simplified output state: the filename is local to initialization, the unused stream-width setting and gzip include were removed, and the relative streamOff/streamStart pair was replaced by one absolute streamOffset.
• Checkpoint resume seeks directly to the saved absolute output position; range-based loops and '\n' replaced index loops and the dependency on mon::lineEnd.
• The continuous-output implementation changed by 50 additions and 89 deletions.

Normal Timed Outputs

• Replaced AgeGroup.h, OutputMeasures.h, info.h, management.h, misc.cpp, mon.cpp, and reporting.h with Monitoring.{h,cpp}, OutMeasures.h, and init.{h,cpp}.
• Consolidated runtime survey state, age groups, cohorts, conditions, measure stores, recording, checkpointing, and output writing under the new Monitoring implementation.
• Replaced the old monitoring enums with Measure, a compact uint16_t index into the single constexpr defs[] table; consteval measure(std::string_view) resolves literals at compile time and rejects unknown names during compilation.
• Every measure now uses its exact XML name in XML, C++, and defs[]; the internalName proxy, measureKey() indirection, and old names such as MHR_HOSTS were removed.
• Removed deployment-method and internal-name metadata from OutMeasure. OutMeasure now contains only the final name, output number, numeric formatting flag, dimensions, and resolved measure index.
• Replaced five dimension booleans with the Dim bitmask for age, cohort, species, genotype, and drug; the temporary dimension operators and hasDim()/clearDim() helpers were removed in favor of direct bit operations.
• Obsolete measure names remain explicitly detectable for scenario error messages but no longer occupy the active measure table; stable output-number gaps remain reserved and must not be reused.
• Stores use one flattened std::variant<std::vector<int>, std::vector<double>> per configured output; integer measures retain compact memory and checkpoint storage while routing and output remain shared.
• Added storesByMeasure routing so one recording reaches every configured store for that measure, including output-number overrides; deployment conditions create an uncategorized, non-output store only when their measure is otherwise disabled.
• Removed the mon::AgeGroup wrapper; humans now store a plain size_t monitoringAgeGroup.
• Survey dates retain both reported and unreported boundaries. Runtime state tracks the current timeline index and caches the next reported survey receiving events, avoiding a timeline scan for every event.
• Replaced approximately fifteen dimension- and type-specific recording functions with three semantic names: recordStat() for the current reported survey, recordEvent() for accumulated events, and recordDeploy() for deployment-method selection.
• All dimensional routing and accumulation now pass through one private recordValue() function; isUsed() checks the direct measure-to-store routing table.
• Unified category validation and custom output-number validation for all measures; negative custom numbers are rejected, and allCauseIMR retains only its special calculation and output path.
• Monitoring report buffers are allocated from scenario-derived dimensions before checkpoint loading. Their serialized lengths must exactly match the expected allocations, and checkpoint data cannot choose an allocation size.
• Monitoring no longer depends on an arbitrary container-length limit; the generic checkpoint-container limit remains 2,000 for containers whose expected sizes are unknown.
• The scenario XML schema, existing XML measure names, stable output numbers, and default text output are unchanged; the monitoring checkpoint layout changed across the overall refactor.

Timed Measures: new Binary Output

• Text and binary formats share the same logical row generation. Text preserves integer/double formatting and may be plain or gzip-compressed.
• Added --output-format txt|bin; text remains the default, exactly one format is written, and either format may be gzip-compressed with --compress-output.
• Output names without an extension automatically receive .txt, .txt.gz, .bin, or .bin.gz; explicit extensions that conflict with the format or compression options are rejected.
• Binary output has a magic value, version, and separate integer and floating-point row counts. It stores contiguous little-endian int32_t survey, category, and measure columns, followed by int32_t integer values and IEEE-754 double floating-point values.
• Grouping binary fields into typed columns reduces storage, improves gzip compression, and allows R to stream each final column directly into a vector and assemble the table by reference.
• Output opening and explicit close are checked so path, write, flush, and close failures return the existing FileIO error status.

Rest Of The Code

• Updated CMake sources and replaced legacy monitoring includes in the simulation, checkpoint, population, clinical, host, within-host, pharmacology, transmission, and intervention code.
• Migrated initialization, survey progression, output writing, checkpointing, and continuous reporting entry points to the new Monitoring APIs.
• Replaced age-group wrapper usage in Human and stored survey-period age and cohort indices directly in Episode where historical event attribution is required.
• Migrated all reporting call sites to exact XML measure names and consistently selected recordStat() for survey statistics and recordEvent() for accumulated events.
• Changed nCMDTReport to recordEvent(), fixing reports before unreported survey markers so they accumulate into the next reported survey; its output-number filtering remains supported.
• Split inoculation reporting into explicit innoculationsPerAgeGroup and innoculationsPerVector calls, including species and genotype dimensions where applicable.
• Deployment call sites now pass final timed and continuous measure pairs directly: nMassIRS/nCtsIRS, nMassGVI/nCtsGVI, nMassITNs/nEPI_ITNs, nMassVaccinations/nEPIVaccinations, nMDAs/nCtsMDA, nMassScreenings/nCtsScreenings, and nMassRecruitOnly/nCtsRecruitOnly; treatment deployments use nTreatDeployments.
• Deploy::Method remains only in the intervention recording API to select timed, continuous, or treatment deployment behavior; it is no longer stored in measure definitions.
• Migrated continuous-output callback registration sites to mon::Continuous::registerCallback().
• Added command-line parsing, validation, defaults, and help text for the survey output format.
• Preserved LF line endings for plain text output on Windows and updated the schema documentation to reference OutMeasures.h.

[b-s-code]

This looks like a strong improvement!

My only blocking query is: what would be involved in adding test coverage of the binary output feature? I assume some change to test/run.py would be needed.

General items:

• Adding binary output is perhaps a bit more than a refactor, and maybe could have been its own PR. No issue including it this time though.
• It looks like continuous output can only be text, not binary? Is this correct?
• Providing an example of how to read a binary output file in R might be useful, either on the PR description, on the wiki, or as an example R script in this repo.

[b-s-code]

This code is a candidate for some cleanup (I assume we don't support live graph any more and the TODO is pre-existing).

Optional.

[b-s-code]

I think this function ought to be renamed.

E.g. notPowerOfTwo(std::pow(2, 30)) returns 1, where I'd expect 0, if I didn't read the implementation.

I can see it doesn't matter much given the enclosing anonymous namespace and context at the call site but I think it would be good to address this while we're already making changes here.

[b-s-code]

Unless the R reader benefits from it, I'd prefer to avoid writing the version number twice.

If writing it twice is useful, I think it would be better to derive one value from the other here, rather than having two literal values.

[b-s-code]

Would userFacingName better describe this?

[b-s-code]

Indentation.