Aries is a 152-GPU cluster, containing AMD MI50 GPUs (courtesy of AMD). It contains 19 GPU nodes (gn01-gn19). Each node contains 8 AMD MI50 GPUs with 32 GB memory each.
Jobs are allocated by nodes, not GPUs. Since there are 8 GPUs per node, each submitted job must be able to use 8 GPUs. Below we provide examples for how to effectively use this resource.
Since each job submission on Aries runs exclusively on an entire GPU node, the user must launch 8 parallel GPU processes. Ideal to have all 8 parallel runs to have similar run time to maximize efficiency. Make sure the 8 parallel runs have different names for the output file generated.
All Aries users should have access to the data storage directory /work/cms16/. Create a sub-directory within /work/cms16/ with your "username".
$mkdir /work/cms16/username
Recommended to store all generated data to your directory in /work/cms16/username and not your /home.
For submitting OpenMM jobs, a singularity container with OpenMM pre-installed is available.
container=/home/pcw2/bin/openmm-ctbp.sif
This container does not have OpenSMOG and needs to be installed with pip3.
$pip3 install OpenSMOG
Usage example with bash submission script, openmm run python script, and input files can be downloaded below.
#!/bin/bash -l #SBATCH --job-name=ctbpexample #SBATCH --nodes=1 #SBATCH --cpus-per-task=96 #set to 96 if not using MPI (OpenMM does not use MPI) #SBATCH --tasks-per-node=1 #SBATCH --export=ALL #SBATCH --mem=0 #each GPU assigned 32 GB by default #SBATCH --gres=gpu:8 #SBATCH --time=1-00:00:00 #max run time is 1 day
Acknowledgments
Please acknowledge the use of this cluster in your publications if you used the cluster:
This work was made possible by the donation of critical hardware and resources from AMD COVID-19 HPC Fund.
The following image can be used for poster and presentations.
