Problem description
Proteus.__init__ resolves its directories through set_directories, which calls create_tmp_folder() in src/proteus/utils/helper.py. That creates $TMPDIR/proteus_<random>/ and nothing anywhere removes it, not at the end of a run and not on failure. A machine that runs many simulations accumulates one directory per run indefinitely.
The construction-time half is the sharper problem. The directory is allocated while the runner is being built, before the configuration has been checked, so a config with a single unrecognised key raises UnknownConfigKeyError out of the constructor with the directory already on disk. No reference to the object escapes __init__, so the caller is left with nothing to clean up with even if it wanted to.
Expected behavior
A run removes its temporary directory when it finishes, and a construction that fails does not leave one behind.
Evidence
Constructing a runner twice against a redirected TMPDIR, once cleanly and once with one unrecognised key in the config:
after clean construction, tmp dirs: 1
raised: UnknownConfigKeyError | Unrecognised configuration key in .../bad.toml
tmp dirs before=1 after=2 (leaked by the failed construction: 1)
The same sequence is visible from the CLI, which announces the directory it has just made and then refuses the config:
[ INFO ] Temporary-file working dir: /var/folders/.../T//proteus_4985149422002/
Error: Unrecognised configuration key in bad.toml:
"not_a_real_key"
My computer
macOS on arm64, where $TMPDIR is a per-user directory the system does not clear promptly, Python 3.12.
Additional notes
Related to #515 but not the same thing. #515 asks for temporary files to live on node-local disk on clusters; this is about the directory never being removed at all, which makes the quota pressure described there strictly worse on a shared node.
A run with params.out.logging = "DEBUG" sets the temporary directory to the output directory and allocates nothing, so only the non-debug path is affected.
Relevant people
@nichollsh
Problem description
Proteus.__init__resolves its directories throughset_directories, which callscreate_tmp_folder()insrc/proteus/utils/helper.py. That creates$TMPDIR/proteus_<random>/and nothing anywhere removes it, not at the end of a run and not on failure. A machine that runs many simulations accumulates one directory per run indefinitely.The construction-time half is the sharper problem. The directory is allocated while the runner is being built, before the configuration has been checked, so a config with a single unrecognised key raises
UnknownConfigKeyErrorout of the constructor with the directory already on disk. No reference to the object escapes__init__, so the caller is left with nothing to clean up with even if it wanted to.Expected behavior
A run removes its temporary directory when it finishes, and a construction that fails does not leave one behind.
Evidence
Constructing a runner twice against a redirected
TMPDIR, once cleanly and once with one unrecognised key in the config:The same sequence is visible from the CLI, which announces the directory it has just made and then refuses the config:
My computer
macOS on arm64, where
$TMPDIRis a per-user directory the system does not clear promptly, Python 3.12.Additional notes
Related to #515 but not the same thing. #515 asks for temporary files to live on node-local disk on clusters; this is about the directory never being removed at all, which makes the quota pressure described there strictly worse on a shared node.
A run with
params.out.logging = "DEBUG"sets the temporary directory to the output directory and allocates nothing, so only the non-debug path is affected.Relevant people
@nichollsh