HOWTO use AmpTools on the JLab farm GPUs
From GlueXWiki
Contents
Access through SLURM
JLab currently provides NVidia Titan RTX or T4 cards on the sciml19 an sciml21 nodes. The nodes can be accessed through SLURM, where N is the number of requested cards (1-4):
>salloc --gres gpu:TitanRTX:N --partition gpu --nodes 1
or
>salloc --gres gpu:T4:N --partition gpu --nodes 1
An interactive shell (e.g. bash) on the node with requested allocation can be opened with srun:
>srun --pty bash
Information about the cards, cuda version and usage is displayed with this command:
>nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 TITAN RTX Off | 00000000:3E:00.0 Off | N/A | | 41% 27C P8 2W / 280W | 0MiB / 24190MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+
AmpTools Compilation with CUDA
This example was done in csh for the Titan RTX cards on sciml1902.
1) Download latest AmpTools release
wget https://github.com/mashephe/AmpTools/archive/refs/tags/v0.12.2.tar.gz
2) Extract files
tar -xvf v0.12.2.tar.gz
3) Load cuda environment module
module add cuda setenv CUDA_INSTALL_PATH /usr/local/cuda
4) Set AMPTOOLS directory
setenv AMPTOOLS $PWD/AmpTools
5) Put root-config in your path
setenv PATH $ROOTSYS/bin:$PATH
6) Edit the AmpTools Makefile to pass the appropriate GPU architecture to the cuda complier (info e.g. here)
CUDA_FLAGS := -m64 -arch=sm_75
7) Build main AmpTools library with GPU support
cd $AMPTOOLS make GPU=1