diff --git a/docs/source/actions.png b/docs/source/actions.png new file mode 100644 index 0000000..f3bdace Binary files /dev/null and b/docs/source/actions.png differ diff --git a/docs/source/developer-guide.rst b/docs/source/developer-guide.rst index 34b986d..2645488 100644 --- a/docs/source/developer-guide.rst +++ b/docs/source/developer-guide.rst @@ -569,4 +569,265 @@ Note that above we link a page from documentation version *21.3.0* into the QUIC Note that you should never link anything from the documentation version named *latest*. This version will change whenever you make changes to the QUICK-docs repository and thus must be used for testing purposes only. -*Last updated by Madu Manathunga on 11/22/2022.* +Maintaining the container +------------------------- +This section provides some guidance on keeping the QUICK container up to date. + +This section assumes that you are familiar with GitHub Actions and Docker containers. +If you are not familiar with both, it is highly recommended that you watch the below videos: + +* `30-minute lesson on GitHub Actions `_ +* `1-hour lesson on Docker `_ + +In addition to those short lessons, the official documentation for each is below: + +* `GitHub Actions Docs `_ +* `Docker Docs `_ + +Updating the Dockerfile +^^^^^^^^^^^^^^^^^^^^^^^ +The Dockerfile may require updates from time to time. +Reasons for updating the Dockerfile could include: updating the CUDA base image, adding a build target for ROCm or accounting for changes in compilation & configuration of QUICK. + +The below description considers the Dockerfile as originally authored, provided below: + +.. code-block:: none + :linenos: + + ############################ + ### Base MPI CUDA 12.0.1 ### + ############################ + FROM nvidia/cuda:12.0.1-devel-ubuntu22.04 AS base-mpi-cuda-12.0.1 + + RUN apt-get update -y \ + && apt-get install -y \ + gfortran \ + cmake \ + g++ \ + openmpi-bin \ + openmpi-common \ + libopenmpi-dev + + RUN mkdir /src \ + && mkdir /src/build + + WORKDIR /src + + # Copy the current version of QUICK into the container + COPY . . + + WORKDIR /src/build + + RUN cmake .. -DCOMPILER=GNU -DCMAKE_INSTALL_PREFIX=$(pwd)/../install -DCUDA=TRUE -DMPI=TRUE + + RUN make -j2 install + + ############################# + ## Runtime MPI CUDA 12.0.1 ## + ############################# + + # Runtime image is smaller than the devel/build image + FROM nvidia/cuda:12.0.1-runtime-ubuntu22.04 AS mpi-cuda-12.0.1 + + RUN apt-get update -y \ + && apt-get install -y \ + openmpi-bin \ + openmpi-common \ + libopenmpi-dev + + # Copy the compiled quick runtimes, leaving behind extra build dependencies & reducing image size + COPY --from=base-mpi-cuda-12.0.1 /src /src + + WORKDIR /src/install + + # Manually run steps from quick.rc + ENV QUICK_INSTALL /src/install + ENV QUICK_BASIS $QUICK_INSTALL/basis + ENV PATH $PATH:$QUICK_INSTALL/bin + +Beginning with line 4, we can see that the base image chosen is "nvidia/cuda:12.0.1-devel-ubuntu22.04" which is aliased as "base-mpi-cuda-12.0.1". +The aliases serve as a shorthand reference inside this Dockerfile, as well as to the Docker build process where these can be specified as build targets. +From the tag ":12.0.1-devel-ubuntu22.04" we can infer that this image is itself based on Ubuntu version 22.04 and that it comes with CUDA version 12.0.1 pre-installed. + +Similarly on line 34, we see an additional image of "nvidia/cuda:12.0.1-runtime-ubuntu22.04" aliased as "mpi-cuda-12.0.1". +The main difference between these two images is the amount of software they have pre-installed. +This translates to their overall file size with the development image having a size of nearly 3.5 GB and the runtime image having a size of nearly 1.25 GB. +A smaller image size is preferrable, as it takes less storage for the end-user as well as less time to download. + +Both images are fetched from the `Docker Hub `_, specifically the repository `nvidia/cuda `_. +Both of these images will require an update in the future to support the latest version of CUDA. +When released, newer versions of these container images can be found on the tags section of the nvida/cuda repo. +These future versions should then be replaced in the Dockerfile. + +Next we consider lines 6-13 and 36-40. We can see additional dependencies being installed for the development and runtime images, respectively. +These are packages required for building and running QUICK that are not included in the respective images. +When additional build and runtime dependencies are added for QUICK these sections should be updated to install those dependencies. + +Continuing with lines 15-27, we can see linux commands being executed according to the directions for compiling QUICK. +When the steps for compiling QUICK change, this section should be updated to match. + +Next on line 43, we see that we copy the /src directory from the development image to the runtime image. +At this point in the process the /src directory has all of the built binaries for QUICK. +This step is critical as it is resposible for drastically reducing the final (runtime) image size from >= 3.5 GB to >= 1.25 GB by leaving behind all of the extra software in the development build. +When the development alias is updated, this line should be updated to match. + +Finally, we see on lines 47-50 that the steps outlined in the quick.rc are performed. +These steps must be executed here so that these environment variables are automatically available to the user each time they run the container. +As steps are added to the quick.rc, they should be added here as well. + +Manual builds with Docker +^^^^^^^^^^^^^^^^^^^^^^^^^ +When making changes to QUICK or the Dockerfile, you can manually build the image to test your changes. +These steps will require that you have an installation of Docker and familiarity with a linux terminal. +These examples assume that you have a `Docker Hub `_ account and that you have QUICK cloned on your computer, referenced as $QUICK_PATH in the examples below. +These examples will reference the Dockerfile discussed above. + +We will cover a few options for building the image: building all stages, building a specific stage, and building without the cache. +Stages, in the context of Docker, means any section of the Dockerfile that begins with the "FROM" keyword until the next instance of "FROM" or the end of the file, whichever comes first. +More can be learned on this subject from the `Docker docs on multi-stage builds `_. + +Building all stages +""""""""""""""""""" +We will start with building all stages, as that is typically the default build choice. +When you build all stages, the last stage in the Dockerfile becomes the final image. + +Begin by navigating to your clone of the QUICK repository: + +.. code-block:: none + + cd $QUICK_PATH + + +To build all stages, run this command from $QUICK_PATH, where the Dockerfile exists: + +.. code-block:: none + + docker build -f Dockerfile -t [your-dockerhub-username]/quick:[your-tag] . + +**Note**: You should replace the brackets and contents with your username and your choice of the tag. + +The build process will take several minutes. When it has finished successfully, you can find the image with this command: + +.. code-block:: none + + docker image ls | grep "quick" + +Assuming the container built successfully, you can then run the container: + +.. code-block:: none + + docker run --rm -it [your-dockerhub-username]/quick:[your-tag] /bin/bash + +When you are done testing the container, you can exit the container: + +.. code-block:: none + + exit + +Building a specific stage +""""""""""""""""""""""""" +This build option is useful when you only need to change something in one stage of the container, for instance installing new runtime dependencies in the runtime stage. +Building just one stage can save you quite a bit of build time, assuming that you have previously built its dependent stages. +In the Dockerfile discussed previously, the runtime stage has a dependency on the base stage, as it copies files from the base stage. + +Begin by navigating to your clone of the QUICK repository: + +.. code-block:: none + + cd $QUICK_PATH + +You can then specify that Docker should only build the stage that you want: + +.. code-block:: none + + docker build -f Dockerfile --target [stage] -t [your-dockerhub-username]/quick:[your-tag] . + +**Note**: You should replace the stage and brackets with an alias from the Dockerfile, for example "mpi-cuda-12.0.1". + +Assuming the container built successfully, you can then run the container: + +.. code-block:: none + + docker run --rm -it [your-dockerhub-username]/quick:[your-tag] /bin/bash + +When you are done testing the container, you can exit the container: + +.. code-block:: none + + exit + +Building without the cache +""""""""""""""""""""""""""" +Lastly, you can force Docker to build a container image without using its cache. +By default, Docker will keep a cache of the layers in a Dockerfile and will only build layers that have been changed or that have a dependency on a changed layer. +You will want to build without the cache when changing things that get copied into the container image, such as the source code for QUICK. + +For example, if you were to change some of the source code, but you don't have to update the Dockerfile in any way, Docker will not know to rebuild the layer and you could find that the container does not reflect your latest changes. +Building without the cache will force Docker to build the entire Dockerfile, which will take longer, but it will catch any changes made outside the Dockerfile. + +Begin by navigating to your clone of the QUICK repository: + +.. code-block:: none + + cd $QUICK_PATH + +You can then specify that Docker should build the entire Dockerfile without the cache: + +.. code-block:: none + + docker build -f Dockerfile --no-cache -t [your-dockerhub-username]/quick:[your-tag] . + +Assuming the container built successfully, you can then run the container: + +.. code-block:: none + + docker run --rm -it [your-dockerhub-username]/quick:[your-tag] /bin/bash + +When you are done testing the container, you can exit the container: + +.. code-block:: none + + exit + +Automated builds with GitHub Actions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The QUICK container should automatically be built and published by the GitHub Actions worfklow "Docker," which is defined in the `GitHub repository `_ at the path .github/workflows/docker-publish.yml. +Runs of this GitHub Action can be found in the repository on the "Actions" tab under the name "Docker." +This workflow is triggered upon pushes and pull requests to the master branch as well as when releases are published. +The workflow builds the Dockerfile located in the root of the repository. +When this workflow runs successfully, the newly built container image can be found on the repository homepage under the packages tab on the right: + +.. image:: packages.png + :width: 300px + :align: center + :height: 275px + :alt: packages + +Clicking "quick" will take you to the page for the container and display the most recent version, as well as other versions: + +.. image:: quick-container.png + :width: 750px + :align: center + :height: 350px + :alt: containers + +From time to time, it may be necessary to update the versions of the steps in the workflow. +When doing so, it is important to leave the first step "Convert IMAGE_NAME to lowercase" in the workflow. +This step conforms the image name, initially "QUICK", to lowercase which is required by Docker for image names. + +Beginning with the second step "Checkout repository", other steps in the workflow will have versions ranging from "@vX" to "@x.x.x" or "@XXXXXXXXXXXXXXX". +For example, at the time of writing, the second step is "actions/checkout@v3". +The step name is in the form of "[author]/[action name][version]". +To update an action, visit the `GitHub Marketplace for actions `_ and enter the name of the action into the search bar. +For example, if we needed to update the second step, we would search the action name of "checkout" and look for the author "actions." + +.. image:: actions.png + :width: 750px + :align: center + :height: 350px + :alt: containers + +Clicking into that first result will take us to the `action's page `_ which will have instructions for using the latest version. +You can then update the version as appropriate in the workflow definition file at the path .github/workflows/docker-publish.yml. + +*Last updated by Kyle Krick on 02/05/2024.* diff --git a/docs/source/installation-guide.rst b/docs/source/installation-guide.rst index d28ff76..3d13a36 100644 --- a/docs/source/installation-guide.rst +++ b/docs/source/installation-guide.rst @@ -139,4 +139,215 @@ Uninstallation and Cleaning Simply delete contents inside build and install directories and / or delete the build and install directories. +Running QUICK in a Container +---------------------------- +For users that are familiar with software containers, QUICK is also made available as a Docker container which can be run in Docker, Singularity and Kubernetes. +The current version of the QUICK container is compiled for x86_64/amd64 CPUs and NVIDIA GPUs and runs serial, MPI, CUDA and CUDA-MPI versions of QUICK. + +**Note:** For CUDA-enabled versions of QUICK, the container requires the host system have the necessary GPU drivers installed. + +**Note:** For MPI-enabled versions of QUICK, the container comes with an MPI runtime which can run on multiple CPUs of a single node. Multi-node execution has not been tested. + +The container is published on the GitHub container registry and can be found in the packages section of the GitHub repository: ``_. +You may want to review the available versions, but for the examples below we will use this image: ghcr.io/merzlab/quick:23.08a-mpi-cuda-12.0.1. + +The following examples have been generalized, so you should be familiar with the system you plan to run the container on. +Many HPC resources will have additional steps that might be required, for example loading modules needed to run Docker or Singularity. + +Below is the file contents for water.in used in the examples below: + +.. code-block:: none + + HF BASIS=cc-pVDZ CUTOFF=1.0d-10 DENSERMS=1.0d-6 ENERGY + + O -0.06756756 -0.31531531 0.00000000 | + H 0.89243244 -0.31531531 0.00000000 | + H -0.38802215 0.58962052 0.00000000 | + +Docker +^^^^^^ +To run the container with Docker, begin by pulling the image: + +.. code-block:: none + + docker image pull ghcr.io/merzlab/quick:23.08a-mpi-cuda-12.0.1 + +Once the image has been pulled, you can run the image and mount your current directory in the container at /data: + +.. code-block:: none + + docker run --rm -it -v $(pwd):/data ghcr.io/merzlab/quick:23.08a-mpi-cuda-12.0.1 /bin/bash + +Change your directory to /data: + +.. code-block:: none + + cd /data + +You can then run quick, quick.MPI, quick.cuda or quick.cuda.MPI on an input file: + +.. code-block:: none + + quick water.in + +When you are done, you can exit the container with the exit command: + +.. code-block:: none + + exit + +Your output file(s) will be left on your local directory: + +.. code-block:: none + + ls + -- + water.in water.out + +Singularity +^^^^^^^^^^^ +To run the container with Singularity, begin by pulling the image: + +.. code-block:: none + + singularity pull docker://ghcr.io/merzlab/quick:23.08a-mpi-cuda-12.0.1 + +This will pull the image and convert the Docker container to a Singularity container such as quick_23.08a-cuda-12.0.1.sif. + +Once the image has been pulled and converted, you can run a shell from your current directory: + +.. code-block:: none + + singularity shell --nv quick_23.08a-cuda-12.0.1.sif + +**Note**: Singularity requires the --nv flag for the CUDA-enabled versions of QUICK + +You can then run quick, quick.MPI, quick.cuda or quick.cuda.MPI on an input file: + +.. code-block:: none + + quick water.in + +When you are done, you can exit the container with the exit command: + +.. code-block:: none + + exit + +Your output file(s) will be left on your local directory: + +.. code-block:: none + + ls + -- + water.in water.out + +Kubernetes +^^^^^^^^^^ +Since the container is published as a Docker container, it can also be run on Kubernetes-based systems. + +Below is an example Kubernetes manifest file, saved as "quick.yaml", for a job: + +.. code-block:: yaml + + apiVersion: batch/v1 + kind: Job + metadata: + name: quick-qm + spec: + template: + spec: + containers: + - name: quick + image: ghcr.io/merzlab/quick:23.08a-mpi-cuda-12.0.1 + command: ["sleep", "infinity"] + resources: + limits: + cpu: "4" + memory: "16Gi" + nvidia.com/gpu: "4" + requests: + cpu: "4" + memory: "16Gi" + nvidia.com/gpu: "4" + restartPolicy: Never + +**Note**: This manifest should be updated to conform to the resources available on the nodes in your Kubernetes cluster. +For example, this manifest assumes the availability of nodes with 4 NVIDIA GPUs. + +Once you have appropriately modified the manifest for your cluster configuration, you can schedule the job: + +.. code-block:: none + + kubectl apply -f quick.yaml + +You can check the availability of the pod running the QUICK container like so: + +.. code-block:: none + + kubectl get pods --watch + --- + NAME READY STATUS RESTARTS AGE + quick-qm-wpwns 0/1 ContainerCreating 0 7s + quick-qm-wpwns 1/1 Running 0 68s + + +Once the pod is running and ready shows (1/1), stop the watch command with ^C (ctrl + c). + +**Note**: The pod name "quick-qm-wpwns" will be different for you. Make sure to update the pod name in the following commands. + +You can now execute a terminal into the container: + +.. code-block:: none + + kubectl exec -it quick-qm-wpwns -- /bin/bash + +You can then copy data into the container via your preffered method. +For simplicity in this example we will use the kubectl cp command from a second terminal on our local machine: + +.. code-block:: none + + kubectl cp water.in quick-qm-wpwns:/root/water.in + +Returning to the terminal attached to the container, we can change directory to /root to find our file: + +.. code-block:: none + + cd /root + ls + --- + water.in + +You can then run quick, quick.MPI, quick.cuda or quick.cuda.MPI on an input file: + +.. code-block:: none + + quick water.in + +When you are done, you can exit the container with the exit command: + +.. code-block:: none + + exit + +You can then copy the output data back to your local machine: + +.. code-block:: none + + kubectl cp quick-qm-wpwns:/root/water.out ./water.out + +You can then verify the transfer succeeded: + +.. code-block:: none + + ls + --- + water.out + +Once you have transferred your data, you should end the job: + +.. code-block:: none + + kubectl delete -f quick.yaml + *Last updated by Madu Manathunga on 11/21/2022.* diff --git a/docs/source/packages.png b/docs/source/packages.png new file mode 100644 index 0000000..e8e71ae Binary files /dev/null and b/docs/source/packages.png differ diff --git a/docs/source/performance.rst b/docs/source/performance.rst index 39d91bc..797767d 100644 --- a/docs/source/performance.rst +++ b/docs/source/performance.rst @@ -37,4 +37,41 @@ The multi-GPU implementation of QUICK is based on MPI. In particular for larger See the following paper for more benchmarks of QUICK multi-GPU version: `Manathunga, M.; Jin, C; Cruzeiro, V.W.D.; Miao, Y.; Mu, D.; Arumugam, K.; Keipert, K.; Aktulga, H.M.; Merz, K.M.; Götz, A.W. Harnessing the Power of Multi-GPU Acceleration into the Quantum Interaction Computational Kernel Program, J. Chem. Theory Comput. 2021, 17, 7, 3955–3966. `_. -*Last updated by Madu Manathunga on 03/03/2022.* +Performance of QUICK Container +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The following benchmark results were obtained using QUICK-23.08a. + +Performance of QUICK Container Compared to Normal Installation +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +This section compares performance between containerized QUICK and a normal installation of QUICK on the same computer using A100 GPUs. + +Both versions of QUICK were timed on the following molecules & settings and were run on one and two GPUs for comparison. + +- Morphine - DFT B3LYP BASIS=6-31G** cutoff=1.0e-7 denserms=1.0e-6 GRADIENT +- PSB5 - DFT B3LYP BASIS=6-31G** cutoff=1.0e-7 denserms=1.0e-6 GRADIENT CHARGE=+1 +- Taxol - DFT B3LYP BASIS=6-31G** cutoff=1.0e-7 denserms=1.0e-6 GRADIENT +- Valinomycin - DFT B3LYP BASIS=6-31G** cutoff=1.0e-7 denserms=1.0e-6 GRADIENT + +The results show the SCF + gradient calculations and total runtime in seconds for each molecule. + +.. image:: quick-container-vs-normal-timings.png + :width: 900px + :align: center + :height: 450px + :alt: quick container versus normal timings on single and dual graphical processing units + +Overall, we found the performance difference to be minimal between containerized QUICK and a normal installation of QUICK. + +Performance of QUICK Container on Various GPUs +"""""""""""""""""""""""""""""""""""""""""""""" +This section compares containerized QUICK performance on various GPUs for the taxol molecule with same settings as above. + +The results show the SCF + gradient calculations and total runtime in seconds on each GPU. + +.. image:: quick-container-taxol-timings-various-gpus.png + :width: 450px + :align: center + :height: 450px + :alt: quick container taxol timings on various graphical processing units + +*Last updated by Kyle Krick on 03/01/2024.* diff --git a/docs/source/quick-container-taxol-timings-various-gpus.png b/docs/source/quick-container-taxol-timings-various-gpus.png new file mode 100644 index 0000000..5697ce2 Binary files /dev/null and b/docs/source/quick-container-taxol-timings-various-gpus.png differ diff --git a/docs/source/quick-container-vs-normal-timings.png b/docs/source/quick-container-vs-normal-timings.png new file mode 100644 index 0000000..a28595c Binary files /dev/null and b/docs/source/quick-container-vs-normal-timings.png differ diff --git a/docs/source/quick-container.png b/docs/source/quick-container.png new file mode 100644 index 0000000..bb2248a Binary files /dev/null and b/docs/source/quick-container.png differ