Downloaded models from huggingface but getting Deepspeed checkpoint loading error

[AndreSlavescu]

Hi, I tried running the premise retrieval pipeline, but ran into this error trace when using the model uploaded on huggingface (https://huggingface.co/kaiyuy/leandojo-lean4-retriever-byt5-small/tree/main):

Enabling DeepSpeed BF16. Model parameters and inputs will be cast to `bfloat16`.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Restoring states from the checkpoint path at /home/aslavescu/projects/ReProver/retriever_model_checkpoints
Traceback (most recent call last):
  File "/home/aslavescu/projects/ReProver/retrieval/main.py", line 25, in <module>
    main()
  File "/home/aslavescu/projects/ReProver/retrieval/main.py", line 20, in main
    cli = CLI(PremiseRetriever, RetrievalDataModule)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 388, in __init__
    self._run_subcommand(self.subcommand)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/cli.py", line 679, in _run_subcommand
    fn(**fn_kwargs)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 864, in predict
    return call._call_and_handle_interrupt(
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 903, in _predict_impl
    results = self._run(model, ckpt_path=ckpt_path)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 974, in _run
    self._checkpoint_connector._restore_modules_and_callbacks(ckpt_path)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 397, in _restore_modules_and_callbacks
    self.resume_start(checkpoint_path)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py", line 79, in resume_start
    loaded_checkpoint = self.trainer.strategy.load_checkpoint(checkpoint_path)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/pytorch_lightning/strategies/deepspeed.py", line 658, in load_checkpoint
    _validate_checkpoint_directory(checkpoint_path)
  File "/home/aslavescu/miniconda3/envs/ReProver/lib/python3.10/site-packages/lightning_fabric/strategies/deepspeed.py", line 898, in _validate_checkpoint_directory
    raise FileNotFoundError(default_message)
FileNotFoundError: The provided path is not a valid DeepSpeed checkpoint: /home/aslavescu/projects/ReProver/retriever_model_checkpoints

Would I need to train the premise retriever to reproduce the results in the README? Or is there a way to use the models listed.

Thanks.

[yangky11]

Could you please list the exact steps for reproducing this error?

[yangky11]

I just updated the evaluation script to use Hugging Face checkpoints instead of PyTorch Lightning checkpoints, which hopefully can eliminate this issue. Please follow the instructions in the new README. Feel free to re-open it if you have other questions.

[zengyuzhi]

Sorry to bother you. If I want to Retrieving Premises for All Proof States, it still face deepspeed checkpoint error. I dont want train a retriver but the provided model checkpoint is only in hugging face format, so I cannot use the huggingface model to retrieve premises for all proof states.