Frequently Asked Questions¶

Building TCHInt¶

Linking a specific BLAS/LAPACK library¶

If you have multiple BLAS/LAPACK implementations installed, you can set the corresponding cmake environment variable to select which to use, e.g., export BLA_VENDOR=Generic before running cmake (other values include Intel10_64_dyn, OpenBLAS, and ATLAS PhiPACK).

On some machines/configurations you may have to pass installation locations to cmake, particularly in order to link the MKL library. You should be able to get the code built including cmake options like the following:

-DBLAS_LIBRARIES="$mkl/libmkl_sequential.so"
-DLAPACK_LIBRARIES="$mkl/libmkl_sequential.so"
-DCMAKE_Fortran_FLAGS="-mkl"

where $mkl is the location of the library files, typically along the lines of .../intel_parallel_studio/2020.4/mkl/lib/intel64.

How to build HDF5 from source¶

It is crucial to match the HDF5 version used by TCHInt with that used by PySCF (as well as by any code calling TCHInt); see hdf5 version conflicts. At this time we suggest installing h5py version 3.0.0:

pip install --user "h5py==3.0.*"

To find out which version of HDF5 is used by h5py you can use:

python3 -c 'import h5py; print(h5py.version.info)' | sed -n "s/^HDF5  *//p"

The following shell commands outline the basic procedure to build a specific version H5VER of HDF5 under INSTALL_DIR:

# Change version and installation directory here.
H5VER="1.12.0"
INSTALL_DIR="$HOME/software/HDF5-$H5VER"

# Fetch, configure, compile, install.
H5MAJ="${H5VER%.*}"
wget "https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-$H5MAJ/hdf5-$H5VER/src/hdf5-$H5VER.tar.bz2"
tar xjvf "hdf5-$H5VER.tar.bz2"
cd "hdf5-$H5VER"
./configure --enable-parallel=yes --enable-fortran=yes --prefix="$INSTALL_DIR"
make -j
make install

Before compiling TCHInt you will need to set HDF5_ROOT to the corresponding installation directory, e.g.:

export HDF5_ROOT="$HOME/software/HDF5-$H5VER"

Building TCHInt with TREXIO support¶

TREXIO is a library offering interoperable data storage for quantum Monte Carlo codes, and can be used in TCHInt for 0/1/2-electron integrals (replacing FCIDUMP and FCIDUMP.bare). Using TREXIO for system data, orbitals, and 3-electron integrals (replacing molden files and TCDUMPs) are planned features.

To build TCHInt with TREXIO support you will need to build TREXIO and install it system-wide, or install it to a user directory and set TREXIO_ROOT to the installation directory prior to compiling TCHInt.

Running TCHInt¶

OpenBLAS problems¶

If you use OpenBLAS you might encounter the following error message at runtime:

BLAS : Program is Terminated. Because you tried to allocate too many memory regions

after which your run is likely to have crashed. To avoid this problem:

export OMP_NUM_THREADS=1

before running TCHInt. If you compile the OpenMP-enabled version of TCHInt you might instead run into the warning message:

OpenBLAS Warning : Detect OpenMP Loop and this application may hang. Please rebuild the library with USE_OPENMP=1 option.

This warning is harmless and can be ignored. See linking BLAS for additional options to use a different library if available.

OpenMPI warnings¶

If you use OpenMPI you may run into warning messages like this one:

--------------------------------------------------------------------------
A process has executed an operation involving a call to the
"fork()" system call to create a child process.  Open MPI is currently
operating in a condition that could result in memory corruption or
other system errors; your job may hang, crash, or produce silent
data corruption.  The use of fork() (or system() or other calls that
create child processes) is strongly discouraged.

The process that invoked fork was:

  Local host:          [[xxxxx,yyyyy],zz] (PID pppppp)

If you are *absolutely sure* that your application will successfully
and correctly survive a call to fork(), you may disable this warning
by setting the mpi_warn_on_fork MCA parameter to 0.
--------------------------------------------------------------------------

This warning is triggered whenever PySCF is invoked from TCHInt (i.e., in all runs except when FCIDUMP and TCDUMP/tcdump.h5 or tcfactors/tcfactors.h5 are present), and can be safely ignored. To suppress the warning message:

export OMPI_MCA_mpi_warn_on_fork=0

before running TCHInt.

HDF5 version conflicts¶

If TCHInt links a different HDF5 version from that linked into h5py the library will throw an error at runtime. It is possible to set an environment variable to suppress the error, but we have verified that PySCF misbehaves in this scenario (returns wrong FCIDUMPs even if TCHInt does not make any calls to the HDF5 library), so the warnings should NOT be ignored.

Developing and debugging TCHInt¶

Code structure¶

The following simplified dependency tree of the main modules in src/ might be helpful to get started with development:

             tchint
       .-----'    '-----.
       |                |
 two_body_term   three_body_term
       |                |
     ukMat         lMat_wrapper
       |             /      \
       '---.    lMat_otf  lMat_precomp
           |     |           |
           |     |   ,-------'
          integral_eval
     .-----'         '-----.
     |                     |
basis_interface        jastrow_mod
     |                 /         \
  (PySCF)   (CASINO Jastrow)  atomic_jastrow

The connections shown are conceptual rather than strictly representing use declarations, but the diagram should more or less show where to find the code for specific tasks. Note that some of the more technical routines are put away in src/support_lib/. The CASINO Jastrow factor implementation lives in casino_gjastrow/src/, while the Python code handling PySCF can be found in python/.

Use of BLAS¶

TCHInt currently passes arrays to BLAS by the first element (Fortran 77 style) because this achieves the fastest possible memory access. For example:

integer :: n=10
real(dp) :: x(n,n), y(n,n)
call dcopy(2*n, x(1,3), 1, y(1,5), 1)

copies the contiguous memory region of size 2*n starting at x(1,3) to the contiguous memory region starting at y(1,5), so this is equivalent to:

y(1:n,5:6) = x(1:n,3:4)

Please leave a working, commented-out version of the code that each BLAS call replaces next to it so as to help others to understand the code. Note that passing Fortran pointers to assumed-size arrays by the first element is typically not allowed by compilers, so we keep BLAS routines external without an explicit interface so that the compiler does not complain.

Two-level MPI¶

TCHInt uses a two-level MPI parallelization scheme in which processes on the same node store some objects as shared-memory arrays, and information is shared between nodes with explicit MPI communication. As a result some codepaths are only triggered in multi-node set-ups. To help with testing and debugging, the split_nodes input keyword can be set to a divisor of the number of processes per node in order to divide each node into split_nodes logical nodes, thus faking a multi-node system – this is used in the integral test for parallel builds.