This page is currently being updated to include examples of submission and configuration commands that can be used on CTBP resources.
It should be noted that this example assumes the use of only one GPU per task and requests an equal amount of memory and CPU resources based on the total resources of each node. The amount of CPU and RAM memory utilized can be increased or decreased based on the user's experience with their system.
Slurm Configuration
NOTS (commons)
This partition includes 16 volta GPU nodes, each equipped with 80 CPUs and 182GB of RAM. In addition, each node includes 2 NVIDIA GPUs.
#SBATCH --account=commons #SBATCH --partition=commons #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=40 #SBATCH --mem=90G #SBATCH --gres=gpu:1 ml gomkl/2021a OpenMM/7.7.0-CUDA-11.4.2
NOTS (ctbp-common)
This partition includes two ampere GPU nodes, each equipped with an AMD EPYC chip featuring 16 CPUs and 512GB of RAM. In addition, each node includes 8 NVIDIA A40 GPUs with 48GB of memory.
#SBATCH --account=ctbp-common #SBATCH --partition=ctbp-common #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=2 #SBATCH --mem=64G #SBATCH --gres=gpu:1 ml gomkl/2021a OpenMM/7.7.0-CUDA-11.4.2
NOTS (ctbp-onuchic)
This partition includes one GPU node, equipped with an AMD EPYC chip featuring 16 CPUs and 512GB of RAM. In addition, each node includes 8 NVIDIA A40 GPUs with 48GB of memory.
#SBATCH --account=ctbp-onuchic #SBATCH --partition=ctbp-onuchic #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=2 #SBATCH --mem=64G #SBATCH --gres=gpu:1 ml gomkl/2021a OpenMM/7.7.0-CUDA-11.4.2
ARIES
This partition includes 19 GPU nodes, each equipped with an AMD EPYC chip featuring 48 CPUs and 512GB of RAM. In addition, each node includes 8 AMD MI50 GPUs with 32 GB of memory each. To submit a job to this queue, it is necessary to launch 8 processes in parallel, each with a similar runtime to minimize waiting time. This ensures that all of the GPUs are used efficiently.
#SBATCH --account=commons #SBATCH --partition=commons #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --export=ALL #SBATCH --gres=gpu:8 module load foss/2020b OpenMM
*Alternative if running 8 jobs in parallel
#SBATCH --account=commons #SBATCH --partition=commons #SBATCH --ntasks=8 #SBATCH --cpus-per-task=6 #SBATCH --threads-per-core=1 #SBATCH --mem-per-cpu=3G #SBATCH --gres=gpu:8 #SBATCH --time=24:00:00 #SBATCH --export=ALL module load foss/2020b OpenMM
PODS
This partition includes 80 GPU nodes, each equipped with an AMD EPYC chip featuring 48 CPUs and 512GB of RAM. In addition, each node includes 8 AMD MI50 GPUs with 32 GB of memory each.
#SBATCH --account=commons #SBATCH --partition=commons #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --export=ALL #SBATCH --gres=gpu module load foss/2020b OpenMM
Remote access to the clusters
To access the servers from outside of Rice it is recommended to connect to them through the gw.crc.rice.edu server. Here we will show how to create a passwordless ssh tunnel that will allow you to securely connect to a remote machine without having to enter a password everytime you connect.
Generate the keys
First generate a pair of public and private keys on your local machine. Open a terminal and enter the following command:
ssh-keygen -t rsa
This will generate a pair of public and private keys, with the default file names id_rsa and id_rsa.pub. Don't share or expose your private key.
Copy the keys to the gateway server
Copy the public key to the remote machine (The gw.crc.rice.edu server). Enter the following command in your terminal, replacing user with the correct rice user id:
ssh-copy-id user@gw.crc.rice.edu
This will copy your public key, id_rsa.pub, to the remote machine and add it to the authorized_keys file on the remote machine.
Test the connection. Enter the following command in your terminal to connect to the remote machine:
ssh user@gw.crc.rice.edu
You should be able to connect to the remote machine without being prompted for a password. Exit to your local machine using Ctrl+D
Create a ssh config file
To make it easier to connect to the remote machine in the future, you can create an ssh config file. This file allows you to specify connection settings and aliases for different remote machines. To create an ssh config file, open a text editor and enter the following information, replacing user_id with your username on the remote machine:
Host crc User user_id HostName gw.crc.rice.edu IdentityFile ~/.ssh/id_rsa Host aries User user_id HostName aries.rice.edu ProxyJump crc Port 22 IdentityFile ~/.ssh/id_rsa Host nots User user_id HostName nots.rice.edu ProxyJump crc Port 22 IdentityFile ~/.ssh/id_rsa
Save the file as ~/.ssh/config.
Connect to the remote machine using the alias. To connect to the remote machine using the alias, enter the following command in your terminal:
ssh crc
This will connect you to the remote machine using the settings and alias specified in the ssh config file. The gateway will be accessible without a password. Exit and try connecting to the compute servers:
ssh nots
The compute server will still ask for a password. To connect to them without a password we will need to copy the keys to them too.
Copy the keys to the compute servers
Copy the public key to the compute servers Enter the following commands in your terminal, replacing user with the correct rice user id:
ssh-copy-id aries ssh-copy-id nots
This will copy your public key, id_rsa.pub, to the remote machine and add it to the authorized_keys file on the remote machine.
Test the connection. Enter the following command in your terminal to connect to the remote machine:
ssh aries
You should be able to connect to the compute servers without being prompted for a password.