fix: restore weights_only=False when reloading best checkpoint

[nileshpatil6]

What does this PR do?

Fixes #15709.

PR #15314 removed the explicit weights_only=False from the torch.load call in NeMoModelCheckpoint.on_save_checkpoint, the code path used when save_best_model=True. Since torch 2.6 changed the default of weights_only to True, this call now fails as soon as the checkpoint's hparams contain an OmegaConf DictConfig, which is the normal case for any Hydra-configured NeMo model. The reporter hit this directly with a real training run and pasted the traceback in the issue.

The checkpoint being loaded here was written to disk by this same training run a few lines above (it is the last/best checkpoint the callback itself just saved), so it is not an untrusted external file, and loading it with weights_only=False is safe. This restores that behavior.

Changelog

• nemo/utils/callbacks/nemo_model_checkpoint.py: pass weights_only=False back to the torch.load call in on_save_checkpoint, with a comment explaining why.

Testing

Ran pytest tests/core/test_exp_manager.py -k test_nemo_checkpoint_save_best_model_1 locally (CPU), it passes with this change. That particular test uses a minimal ExampleModel whose hparams do not include a DictConfig, so it passes both before and after this change and does not by itself catch the regression, the regression only shows up once hparams contain an OmegaConf object, as in the issue's traceback.

I was not able to fully set up NeMo's heavier ASR/TTS dependency stack in this environment to add a new end to end regression test in reasonable time. I verified the fix by reading the exact code path (this is the only torch.load call in this method, loading a file this same callback just wrote), comparing against the pre-#15314 behavior, and confirming the diff is syntactically valid and mirrors the exact change #15314 made in reverse for this one call site.

Usage

No API change, this only affects the internal reload of the best checkpoint when save_best_model=True in ModelCheckpoint params.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.