A bad reference config fails every inference worker with no reason given

[timlichtenberg]

Problem description

proteus infer checks its reference config with os.path.isfile and nothing more (src/proteus/inference/inference.py, the block that resolves config['ref_config']). The file is copied into the output directory and handed to the workers without ever being read against the schema, so a mistake in it survives the whole setup.

The first worker is where it surfaces, and it surfaces badly. run_proteus in src/proteus/inference/objective.py writes the per-case config out of the reference file and runs proteus start as a subprocess with stdout=subprocess.DEVNULL and stderr=subprocess.STDOUT. The child does refuse an unrecognised key and does say which one, but it says it on stderr, and stderr follows stdout into /dev/null. What reaches the log is Failed to run PROTEUS for worker=0 iter=0; exit code 1, with nothing pointing at the reference config. Every worker then fails the same way, so a whole optimisation burns through its budget with no usable diagnosis.

The discarded output is not specific to this fault. Any reason a child run dies is thrown away the same way.

Expected behavior

The reference config is loaded through read_config_object at startup, next to the existing os.path.isfile check, so a typo stops the run before any worker is spawned and names the offending key.

Separately, when a child run fails, its output is captured and logged rather than discarded.

Evidence

The child reports the fault, and reports it exactly where the inference module has arranged not to look:

$ proteus start -c bad.toml --offline ; echo "exit=$?"
Error: Unrecognised configuration key in bad.toml:
  "not_a_real_key"
  This key is unrecognised, so setting it will have no effect. Check for typos or outdated option names.
exit=1

bad.toml is input/dummy.toml with one unrecognised key added. That message is on stderr, which run_proteus sends to /dev/null.

My computer

macOS on arm64, Python 3.12.

Additional notes

Two independent fixes here. Validating the reference config up front is the one that closes this specific hole. Capturing the child's output is worth doing on its own, since it applies to every way a worker run can fail.

Relevant people

@nichollsh

[nichollsh]

Agreed with both issues: (i) the config should be validated near the start of a proteus infer call to avoid errors being thrown later, and (ii) worker outputs should be captured robustly.

However, I think this also relates to a wider problem about how proteus infer's ABO scheme dispatches workers. Calling them as subprocesses is far from optimal, since it means each case gets its own Python/Julia instance. Maybe this is an opportunity to re-visit how proteus infer dispatches workers, which would save CPU-hours and memory. I found that the start-up time of PROTEUS is actually one of the largest resource drains in an ABO run. @egpbos and @stuitje ?

[egpbos]

Not sure if this answers your question (I'm on holiday now and just briefly skimmed while the kids are playing ;) responding mostly to "I found that the start-up time of PROTEUS is actually one of the largest resource drains in an ABO run"), but I was thinking earlier about whether we can somehow transform the first phase of PROTEUS into a separate initial conditions generator that saves IC files that could then be ingested by the remaining run. This makes most sense if those ICs can be used multiple times (which would be the case if the ICs only need a subset of parameters that are constant to the sweeps you want to run, i.e. when you only need one IC for N runs), but I haven't checked yet whether that's actually something that ever happens in real cases.

[stuitje]

Apologies for the late comment: I am back at the university since this week and now starting up work again + beginning my msc thesis. I think I can definitely take up this issue as part of my thesis.

I agree that the config validation is a high-priority issue, and from what I have seen of the code so far + my single inference run before the summer, I also agree with @nichollsh that it would be a good idea to reconsider the subprocess design. If every subprocess has to startup Julia from scratch it likely eats up a lot of time. However, since I would like to take some time to get more familiar with the code and how it runs, and to think about a way to dispatch workers more efficiently without causing different issues, my suggestion is to:

1. fix the config validation problem, soon
2. capture worker outputs more robustly, after
3. redesign the worker dispatchment; later, after EPSC