Skip to content

get_stats crashes when a label is all-NaN or missing #2

Description

@sirmarcel

Problem

evaluate.metrics.get_stats() crashes when a requested key has no valid (non-NaN) data across any sample in the dataset.

Failure 1: all-NaN labels

If every sample has NaN for a given key (e.g. stress is missing for the whole dataset but "stress" is in keys), tmp[key] remains an empty list. np.concatenate([]) then raises:

ValueError: need at least one array to concatenate

Failure 2: missing key

If a key is requested but doesn't exist in sample.labels at all (not NaN, just absent), the label lookup raises a KeyError.

Reproducer

playground/repro_get_stats_nan.py:

import numpy as np
from collections import namedtuple
from marathon.evaluate.metrics import get_stats

Sample = namedtuple("Sample", ("structure", "labels"))

samples = [
    Sample(
        {"positions": np.zeros((3, 3))},
        {
            "energy": 1.0,
            "forces": np.ones((3, 3)),
            "stress": float("nan") * np.ones((3, 3)),
            "num_atoms": 3,
        },
    ),
]

# Crashes — stress is all-NaN:
stats = get_stats(samples, keys=["energy", "forces", "stress"])

Where

marathon/evaluate/metrics.py, get_stats(), lines ~120–154.

Suggested fix

  • Skip keys with no valid data: omit from returned stats or fill with NaN sentinels.
  • Guard label lookup with key in l check (treat missing same as NaN).
  • Downstream (get_metrics_fn, emit) needs to handle absent stats gracefully (skip R² when unavailable).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions