Compiling VASP
Overview
The Vienna Ab initio Simulation Package (VASP), is a package for performing ab initio quantum mechanical calculations using either Vanderbilt pseudopotentials, or the projector augmented wave method and a plane wave basis set.
- VASP Homepage: Access to VASP News, Wiki, Support Forum and Portal
- VASP Wiki: Menu
- Please consult the VASP FAQ for more details regards licensing.
For Example: To whom are VASP licenses given?
VASP licenses are given to well-defined research groups only. They are not department-or institution-wide, also they are not personal licenses. All members of a VASP-group have to work in the same organizational unit (Department, Institute) at the same location.
VASP-licenses are not transferable from one research institution to another one. Any transfer of a license to another group at the same research institution has to be approved by the VASP Software GmbH explicitly.
Using
Note
VASP is NOT publicly available to users across the clusters. Instead, there are local installations per project with each project having its own license agreement. If ARCC comes across users abusing the licensing agreement we will have to act on it.
Multicore
Note: Although ARCC can assist in the functional setup and running of VASP on the cluster, we do leave the science up to the researchers.
With this in mind, there are a number of pages on the VASP Wiki that make suggestions on how best to configure simulation inputs with respect to nodes/cores, as well as memory considerations.
VASP Wiki:
- NCORE: determines the number of compute cores that work on an individual orbital. Default: NCORE = 1
- NPAR determines the number of bands that are treated in parallel. Default: NPAR = number of cores
- Performance issues, try NCORE, KPAR, ALGO, LREAL
- Not enough memory
- Memory requirements
- These are just a few, please look through the VASP Wiki for others. If you find further details, please pass them onto us and we can add them to this page.
- ISTART: Related to starting a job and determines whether or not to read the WAVECAR file.
Others:
- Yale: CRC: VASP
- NSC: Selecting the Right Number of Cores for a VASP Calculation
Installation Guidelines
The basic process is to unpack the provided source tar file. Copy and update an appropriate arch/makefile.include."version" file, and then build.
Below is an example using a version that ARCC believes is suitable. By all means select and try a different version but you will need to select appropriate compilers and related libraries, as well as updating the makefile.include.
With the example below, you will need to update project-name appropriately.
MedicineBow: Version 6.3.2
General Process:
# We suggest building this software underneath your related project, so it can be shared across that project,
# specifically under /project/<project-name>/software/.
[]$ tar -xf vasp.6.3.2.tar
[]$ cd vasp.6.3.2/
[]$ cp arch/makefile.include.gnu_omp .
[]$ mv makefile.include.gnu_omp makefile.include
# Update the makefile.include file.
[]$ module load gcc/14.2.0 openmpi/5.0.5 fftw/3.3.10-ompi openblas/0.3.24 netlib-scalapack/2.2.0-ompi wannier90/3.1.0 hdf5/1.14.3__hl_True__fortran_True-ompi
[]$ make all
[]$ ls bin/
vasp_gam vasp_ncl vasp_std
You need to update the makefile.include file. The following modifications have been made:
- Commented out native compiling.
- Have explicitly added the paths to openblas, scalapack and fftw.
- Building with, and thus adding paths to, hdf5 and wannier90.
- The wannier90 library typically installs the actual library file in its parent folder, and not a sub lib/ folder. This modifies the makefile.include to: LLIBS += -L$(WANNIER90_ROOT)/ -lwannier i.e the lib folder is not included.
- Compiling fftlib.
makefile.include
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxGNU\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dfock_dblbuf \
-D_OPENMP
CPP = gcc -E -C -w $*$(FUFFIX) >$*$(SUFFIX) $(CPP_OPTIONS)
FC = mpif90 -fopenmp
FCL = mpif90 -fopenmp
FREE = -ffree-form -ffree-line-length-none
FFLAGS = -w -ffpe-summary=none
OFLAG = -O2
OFLAG_IN = $(OFLAG)
DEBUG = -O0
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
OBJECTS_O1 += fftw3d.o fftmpi.o fftmpiw.o
OBJECTS_O2 += fft3dlib.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = $(FC)
CC_LIB = gcc
CFLAGS_LIB = -O
FFLAGS_LIB = -O1
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = g++
LLIBS = -lstdc++
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself, change this to the
# relevant target when cross-compiling for another architecture
# VASP_TARGET_CPU ?= -march=native
# FFLAGS += $(VASP_TARGET_CPU)
# For gcc-10 and higher (comment out for older versions)
FFLAGS += -fallow-argument-mismatch
# BLAS and LAPACK (mandatory)
OPENBLAS_ROOT ?= /apps/u/spack/gcc/14.2.0/openblas/0.3.24-52ip5la/
BLASPACK = -L$(OPENBLAS_ROOT)/lib -lopenblas
# scaLAPACK (mandatory)
SCALAPACK_ROOT ?= /apps/u/spack/gcc/14.2.0/netlib-scalapack/2.2.0-dbchdg6/
SCALAPACK = -L$(SCALAPACK_ROOT)/lib -lscalapack
LLIBS += $(SCALAPACK) $(BLASPACK)
# FFTW (mandatory)
FFTW_ROOT ?= /apps/u/spack/gcc/14.2.0/fftw/3.3.10-mk3sml6/
LLIBS += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp
INCS += -I$(FFTW_ROOT)/include
# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT ?= /apps/u/spack/gcc/14.2.0/hdf5/1.14.3-2frri7c/
LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
CPP_OPTIONS += -DVASP2WANNIER90
WANNIER90_ROOT ?= /apps/u/opt/gcc/14.2.0/wannier90/3.1.0/
LLIBS += -L$(WANNIER90_ROOT)/ -lwannier
# For the fftlib library (recommended)
CPP_OPTIONS+= -Dsysv
FCL += fftlib.o
CXX_FFTLIB = g++ -fopenmp -std=c++11 -DFFTLIB_THREADSAFE
INCS_FFTLIB = -I./include -I$(FFTW_ROOT)/include
LIBS += fftlib
LLIBS += -ldl
Caution
Note: If you wish to use a different make.include.your_choice or try another process, then you’ll need to module load potentially a different compiler, and build core libraries if not built for that chosen compiler.
Suggested Module File Creation
If you have installed under say: /project/project-name/software/vasp.6.3.2/ you could create the following:
Module file creation process:
6.3.2.lua
whatis([[Name : vasp]])
whatis([[Version : 6.3.2]])
whatis([[Short description : The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles. ]])
help([[
vasp : 6.3.2
The Vienna Ab initio Simulation Package (VASP) is a computer program for atomic scale materials modelling, e.g. electronic structure calculations and quantum-mechanical molecular dynamics, from first principles.
https://vasp.at/
]])
if not isloaded("gcc/14.2.0") then
load("gcc/14.2.0")
end
if not isloaded("openmpi/5.0.5") then
load("openmpi/5.0.5")
end
if not isloaded("fftw/3.3.10-ompi") then
load("fftw/3.3.10-ompi")
end
if not isloaded("openblas/0.3.24") then
load("openblas/0.3.24")
end
if not isloaded("netlib-scalapack/2.2.0-ompi") then
load("netlib-scalapack/2.2.0-ompi")
end
if not isloaded("wannier90/3.1.0") then
load("wannier90/3.1.0")
end
if not isloaded("hdf5/1.14.3__hl_True__fortran_True-ompi") then
load("hdf5/1.14.3__hl_True__fortran_True-ompi")
end
prepend_path("PATH", "/project/<project-name>/software/vasp.6.3.2/bin")
To use module file:
Installation: Potential Issues
Bin Folder Executables: Permission Denied If you get an error of the form:
Once built, the files in the bin folder should be executable. Check they have the x permission defined for the groups:[bin]$ ls -al
-rwxrwxr-x 1 <username> <project-name> 19190336 Oct 7 14:32 vasp_gam
-rwxrwxr-x 1 <username> <project-name> 19547024 Oct 7 14:39 vasp_ncl
-rwxrwxr-x 1 <username> <project-name> 19481464 Oct 7 14:25 vasp_std
MedicineBow: Version 6.3.2: GPU
Here is the process to build a version that utilizes GPUs.
This example uses the nvidia hpc-sdk to build, and subsequently should use its provided version of mpiexec (not srun).
Example Build Process
[~]$ cd /project/<project>/software/
[software]$ mkdir vasp
[software]$ cd vasp
[vasp]$ mkdir vasp.6.3.2.gpu
[vasp]$ cp vasp.6.3.2.tar vasp.6.3.2.gpu/
[vasp]$ cd vasp.6.3.2.gpu/
[vasp.6.3.2.gpu]$ cd vasp.6.3.2/
[vasp.6.3.2]$ mv * ..
[vasp.6.3.2]$ cd ..
[vasp.6.3.2.gpu]$ rmdir vasp.6.3.2/
[vasp.6.3.2.gpu]$ rm vasp.6.3.2.tar
# Setup makefile.include
# Based on arch/makefile.include.nvhpc_ompi_mkl_omp_acc
[vasp.6.3.2.gpu]$ vim makefile.include
# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
-DMPI -DMPI_BLOCK=8000 -Duse_collective \
-DscaLAPACK \
-DCACHE_SIZE=4000 \
-Davoidalloc \
-Dvasp6 \
-Duse_bse_te \
-Dtbdyn \
-Dqd_emulate \
-Dfock_dblbuf \
-D_OPENMP \
-D_OPENACC \
-DUSENCCL -DUSENCCLP2P
CPP = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX) > $*$(SUFFIX)
# N.B.: you might need to change the cuda-version here
# to one that comes with your NVIDIA-HPC SDK
# FC = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp
# FCL = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.0 -mp -c++libs
FC = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp
FCL = mpif90 -acc -gpu=cc70,cc80,cuda12.3 -mp -c++libs
FREE = -Mfree
FFLAGS = -Mbackslash -Mlarge_arrays
OFLAG = -fast
DEBUG = -Mfree -O0 -traceback
OBJECTS = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o
LLIBS = -cudalib=cublas,cusolver,cufft,nccl -cuda
# Redefine the standard list of O1 and O2 objects
SOURCE_O1 := pade_fit.o
SOURCE_O2 := pead.o
# For what used to be vasp.5.lib
CPP_LIB = $(CPP)
FC_LIB = nvfortran
CC_LIB = nvc -w
CFLAGS_LIB = -O
FFLAGS_LIB = -O1 -Mfixed
FREE_LIB = $(FREE)
OBJECTS_LIB = linpack_double.o
# For the parser library
CXX_PARS = nvc++ --no_warnings
##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself , change this to the
# relevant target when cross-compiling for another architecture
# VASP_TARGET_CPU ?= -tp host
# FFLAGS += $(VASP_TARGET_CPU)
# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')
# If the above fails, then NVROOT needs to be set manually
#NVHPC ?= /opt/nvidia/hpc_sdk
#NVVERSION = 21.11
#NVROOT = $(NVHPC)/Linux_x86_64/$(NVVERSION)
## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2
# OFLAG_IN = -fast -Mwarperf
# SOURCE_IN := nonlr.o
# Software emulation of quadruple precsion (mandatory)
QD ?= $(NVROOT)/compilers/extras/qd
LLIBS += -L$(QD)/lib -lqdmod -lqd
INCS += -I$(QD)/include/qd
# Intel MKL for FFTW, BLAS, LAPACK, and scaLAPACK
# MKLROOT ?= /path/to/your/mkl/installation
MKLROOT ?= /apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2
# LLIBS_MKL = -Mmkl -L$(MKLROOT)/lib/intel64 -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
# MKLLIBS = -Mmkl
MKLLIBS = -lmkl_intel_lp64 -lmkl_pgi_thread -lmkl_core -pgf90libs -mp -lpthread -lm -ldl
LLIBS_MKL = -L$(MKLROOT)/lib -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 $(MKLLIBS)
INCS += -I$(MKLROOT)/include/fftw
# Use a separate scaLAPACK installation (optional but recommended in combination with OpenMPI)
# Comment out the two lines below if you want to use scaLAPACK from MKL instead
# SCALAPACK_ROOT ?= /path/to/your/scalapack/installation
# LLIBS_MKL = -L$(SCALAPACK_ROOT)/lib -lscalapack -Mmkl
LLIBS += $(LLIBS_MKL)
# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT ?= /path/to/your/hdf5/installation
#LLIBS += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS += -L$(WANNIER90_ROOT)/lib -lwannier
# For the fftlib library (hardly any benefit for the OpenACC GPU port, especially in combination with MKL's FFTs)
#CPP_OPTIONS+= -Dsysv
#FCL += fftlib.o
#CXX_FFTLIB = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_USE_MKL -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(MKLROOT)/include/fftw
#LIBS += fftlib
#LLIBS += -ldl
#- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Build VASP:
[vasp.6.3.2.gpu]$ module load nvhpc-sdk/24.3
[vasp.6.3.2.gpu]$ module load nvhpc-openmpi3/24.3
[vasp.6.3.2.gpu]$ export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2
[vasp.6.3.2.gpu]$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/
[vasp.6.3.2.gpu]$ make all
if [ ! -d build/std ] ; then mkdir -p build/std ; fi
cp src/makefile src/.objects src/makedeps.awk makefile.include build/std
make -C build/std VERSION=std check
...
make[2]: Leaving directory '/cluster/medbow/project/design-lab/software/VASP/vasp.6.3.2.gpu/build/ncl'
make[1]: Leaving directory '/cluster/medbow/project/design-lab/software/VASP/vasp.6.3.2.gpu/build/ncl'
[vasp.6.3.2.gpu]$ ls bin/
vasp_gam vasp_ncl vasp_std
# Change permissions to allow rest of project to use:
[vasp.6.3.2.gpu]$ cd ..
[vasp]$ chown -R <username>:<project> vasp.6.3.2.gpu
Example submission scripts: You will need to set project and various options such as time/memory/cpus-per-task as required.
Single Compute Node using an a30
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --time=8:00:00
#SBATCH --account=<project>
#SBATCH --partition=mb-a30
#SBATCH -G 1
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
module load nvhpc-sdk/24.3
module load nvhpc-openmpi3/24.3
export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/
mpiexec -np 1 /project/<project>/software/VASP/vasp.6.3.2.gpu/bin/vasp_std
Two Compute Nodes using an H100
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8
#SBATCH --time=8:00:00
#SBATCH --account=<project>
#SBATCH --partition=mb-a30
#SBATCH -G 2
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
module load nvhpc-sdk/24.3
module load nvhpc-openmpi3/24.3
export MKLROOT=/apps/u/opt/compilers/oneapi/2024.2.1/mkl/2024.2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MKLROOT/lib:/apps/u/spack/gcc/11.4.1/gcc/11.4.0-k3wm6wb/lib64/
mpiexec -np 2 /project/<project>/software/VASP/vasp.6.3.2.gpu/bin/vasp_std