Add benchmarks of selected models and architectures - #2198
Conversation
frostedoyster
added
documentation
| @@ -0,0 +1,67 @@ | |||
| # OMol force accuracy and runtime | |||
|
|
|||
|  | |||
There was a problem hiding this comment.
should we add a note that you can also run the script for energy benchmark if they want?
There was a problem hiding this comment.
Unfortunately the script we're including only does force MAEs. Energy MAEs hit a problem with MACE-MH-1 because it uses different energy references, so the energy MAEs are very much off for that model
There was a problem hiding this comment.
i think thats fine, maybe just make a note?
There was a problem hiding this comment.
Yes, I will add a note on the internal readme of the benchmark (benchmarks/omol/README.md)
| > :warning: **FAIRChem version 2 is a breaking change from version 1 and is not compatible with our previous pretrained models and code.** | ||
| > If you want to use an older model or code from version 1 you will need to install [version 1](https://pypi.org/project/fairchem-core/1.10.0/), | ||
| > as detailed [here](#looking-for-fairchem-v1-models-and-code). | ||
| ## UMA is now fast! |
There was a problem hiding this comment.
UMA is now much faster!
|
|
||
| [](https://facebook-fairchem-uma-demo.hf.space/) | ||
|
|
||
| ## Legacy models |
There was a problem hiding this comment.
Yeah, I'm pretty sure AIs will be able to find the v1 installation instructions if anyone is interested in running the old models, removing now
| [benchmark details and reproduction scripts](benchmarks/omol/README.md) | ||
| for the full setup. | ||
|
|
||
| ## Latest news |
There was a problem hiding this comment.
add a line to latest news for the speed up here (point to the right fairchem version), also add enzyme paper :)
|
|
||
| The vertical axis is force MAE (meV/Å) on the public OMol25 validation set; the | ||
| horizontal axis is mean runtime (ms/step) for an ASE NVE step on a water system | ||
| using an NVIDIA H200. See the |
There was a problem hiding this comment.
should we have a link to point to omol HF leaderboard?
2 participants
This PR adds Python- and ASE-based evaluation of a few selected models measuring their accuracy on the OMol validation set and speed on Nvidia H200 GPUs