- The Cray programming environment on the XC50 platform
- Building code that depends on external libraries
- Building code on the CS500 platform
You will learn:
- how to compile code (Fortran, C, C++) using the Cray programming environment on the XC50 platform
- how to build code that depends on external libraries
The example programs used in this lesson can be found in the _code directory of the “hpc_training” repository.
The Cray programming environment on the XC50 platform
Building Fortran, C, or C++ code on the XC50 platform requires using the Cray programming environment. From a user perspective, the programming environment consists of a set of environment modules that select a compiler, essential libraries such as the MPI library, a CPU target, and more. The build process on the XC50 thus differs slightly from a standard Linux system. Non-compiled code, such as Python or R programs, do not use the programming environment. Note, however, that loading a module provided by NeSI/NIWA to get access to, e.g., the “RegCM” code, may change the Cray programming environment in that Cray environment modules may be swapped.
- It is essential to use a Programming Environment (PrgEnv-cray, PrgEnv-intel or PrgEnv-gnu) when you build code, otherwise it is very likely that problems at build time or run time appear
- Never use
module purgeon the XC50 platform, this will render the programming environment unusable, and you will have to log out and log back in
- Code that was built on the XC50 platform is unlikely to run on Māui’s CS500 platform or on Mahuika’s CS400 platform; please rebuild your code when you change platform
The build node
Māui has a dedicated build node,
login.maui.niwa.co.nz, which should be used for building code. Please do not build code on the compute nodes by submitting a build job through SLURM:
- The compute nodes only run a thin operating system with very few command line utilities, it is thus likely that your build will fail
- The file system on XC50 compute nodes is optimised for handling large block IO, small block IO that is typical for a build job is inefficient
- Submitting a job will allocate entire nodes. This is a waste of compute resources, especially if only one core or a few cores are used
Furthermore, please keep in mind that the build node is a shared resource. Instead of using as many parallel build processes as possible (with
make -j), please limit the amount of processes (
make -j 5 for example).
Choosing a programming environment
The following Programming Environments are provided on Māui, named after the underlying compiler suite:
PrgEnv-cray environment is the default. If you want to change programming environment to use the Intel or GNU compilers, run
module swap PrgEnv-cray PrgEnv-intel
module swap PrgEnv-cray PrgEnv-gnu
Note that several compiler versions are currently installed, in case of GNU for example:
> module avail gcc -------------------------------------- /opt/modulefiles -------------------------------------- gcc/4.9.3 gcc/5.3.0 gcc/6.1.0 gcc/7.1.0 gcc/7.3.0(default)
To change GCC version, run for example
module swap gcc gcc/7.1.0
GCC v6.1.0 or later is required to build code that can make use of the Intel Skylake microarchitecture and its advanced capabilities, such as AVX-512, on the XC50 platform.
Note: There is not the best compiler. Depending on your application/algorithms, different compilers can optimise the code better. Keep in mind trying different compilers.
Targetting a CPU
Compiling a program translates source code into machine instructions. It is important to let the compiler know for which CPU (“target”) the executable shall be build, to make best use of that CPU’s capabilities. Māui uses Intel Skylake microprocessors on all XC50 build and compute nodes, which come with AVX-512 vector instructions, enabling better performance for some codes.
CPU targets can be set by loading a module. By default, module
craype-x86-skylake is loaded. In the rare case that you encounter problems with the Skylake target at build time or run time, try target for “Broadwell” processors instead:
module swap craype-x86-skylake craype-broadwell
Choosing the “Broadwell” target is also necessary if you want to build code using the older GCC compilers prior to GCC v6.1.0, which were released before Skylake became available. If you see the error message
craype-x86-skylake requires cce/8.6 or later, intel/15.1 or later, or gcc/6.1 or later
when trying to swap to the
PrgEnv-gnu environment, or an error message of the kind
f951: error: bad value (skylake-avx512) for -march= switch
when you compile a program with a GNU compiler, run
module swap craype-x86-skylake craype-broadwell
and try again.
Make sure that a target is always set - otherwise the compilers will produce generic code that runs on many processors of the “x86-64” family, and the program will thus not be able to benefit from capabilities such as AVX-512. You will see the following warning message when you run a compiler:
No supported cpu target is set, CRAY_CPU_TARGET=x86-64 will be used. Load a valid targeting module or set CRAY_CPU_TARGET
Using the compiler drivers
The programming environment provides compiler drivers for compiling Fortran, C, and C++ code. This means that you will need to use the following commands instead of the actual compilers:
ftn -o simpleMpi simpleMpi.f90 # compile Fortran code cc -o simpleMpi simpleMpi.c # compile C code CC -o simpleMpi simpleMpi.cxx # compile C++ code
The drivers will ensure correct linking of your code with compiler runtime libraries, and with Cray-supported libraries (such as Cray’s “libsci” scientific library, or Cray’s version of netCDF). It is therefore not recommended to use the compilers directly, there is a good chance that the executable will fail to build or run correctly.
The compiler drivers automatically add necessary compile and link flags to the compile/link line for the selected hardware and Cray-supported libraries. If you are interested in seeing what the compiler driver does, add the
ftn -craype-verbose -o simpleMpi simpleMpi.f90
Further compiler driver options can be found on their man pages:
man ftn man cc man CC
The compiler drivers will also automatically build MPI codes correctly, there is no need to use special compilers or add additional compiler or linker flags.
Note that running an MPI code on the build node (
will fail with an error message, as there is no MPI runtime environment:
[Wed Oct 18 02:00:14 2017] [c0-0c0s3n1] Fatal error in MPI_Init: Other MPI error, error stack: MPIR_Init_thread(537): MPID_Init(247).......: channel initialization failed MPID_Init(636).......: PMI2 init failed: 1
If you want to run a short test of your build, use SLURM’s
mpiexec command that submits your program to a compute node on the fly, e.g.,
SLURM_PARTITION=nesi_research mpiexec -n 6 simpleMPI
Common compiler options
Although the compiler drivers
CC have a few options of their own, such as the
-craype-verbose flag, they will pass through any additional compiler options to the underlying compiler. This means that you will still need to choose compiler flags that are specific to the Cray, Intel, or GNU compilers, and you will need to change them if you decide to switch compilers.
For example, if you wanted to use the gfortran compiler, activate compiler warnings (
-Wall), and require aggressive compiler optimisation (
-O3), you would use the following commands:
module swap PrgEnv-cray PrgEnv-gnu ftn -Wall -O3 -o simpleMpi simpleMpi.f90
The following table provides a list of commonly used compiler options:
||Set level of debugging information, some levels may disable certain compiler optimisations|
|Light compiler optimisation||
|Aggressive compiler optimisation||
||This may affect numerical accuracy|
Additional compiler options are documented on the compiler man pages, which are accessible after loading the corresponding programming environment:
|Fortran||man crayftn||man ifort||man gfortran|
|C||man craycc||man icc||man gcc|
|C++||man crayCC||man icpc||man g++|
The man pages are often largely incomplete, further documentation can be found online:
- Cray Compiler Environment: Cray Fortran v8.7, Cray C and C++ v8.7
- Intel compilers: Intel Fortran Compiler v17.0, Intel C and C++ Compiler v17.0
- GNU compilers: GCC C and C++ v4.9.4, GCC C and C++ v7.1.0, GNU Fortran v4.9.4, GNU Fortran v7.1
Building code that depends on external libraries
While linking external libraries, one need to pay attention to the correct compiler and linker setup. This, depends on the correct library version (working properly with the compiler and the link type) and the used link options. These depend on whether the libraries have been provided by Cray, by NeSI/NIWA, or if you built them yourself.
Many libraries are provided in modules. You can search them using
and look in the module description using:
module help <module-name>
Sometimes modules provide multiple libraries, e.g. cray-libsci.
Using libraries provided by Cray
If a library has been provided by Cray, the compiler drivers will automatically take care of adding search paths for include files and libraries, and they will add the library names to the linker line. For example, to build a program that uses the netCDF library provided by the
cray-netcdf module, run the commands
module load cray-netcdf ftn -o simple_xy_wr simple_xy_wr.f90
Keep in mind that such automatic treatment of dependencies will only work if the libraries have been provided by Cray - you can recognise those by their module name, which always starts with
Note also that correct versions of the libraries (Cray CCE, Intel, or GNU builds) will automatically be used after swapping programming environment. This is particularly important for libraries that provide Fortran 90 modules, due to their compiler-specific format.
Using libraries provided by NeSI/NIWA
The situation is different when you use a library that is provided by NeSI/NIWA. They can be recognised by the
CrayGNU suffix attached to their version number. In this case, you will have to provide search paths using the
-I flag for include files, and
-L for library files, and the library names have to be explicitly added to the linker line. Libraries are not always provided for all compiler suites and versions.
Note that library names are specified in a specifically formatted form,
-l<library name>. The linker then expects to find a library file named
lib<library name>.a (for a static library) or
lib<library name>.so (for a shared library), e.g.,
libnetcdf.a. Note that you may need to list several libraries to link successfully, e.g.,
-lA -lB for linking against libraries “A” and “B”. The order in which you list libraries matters, as the linker will go through the list in order of appearance. If library “A” depends on library “B”, specifying
-lA -lB will work. If library “B” depends on “A”, use
-lB -lA. If they depend on each other, use
-lA -lB -lA (although such cases are quite rare).
Consider the following example where the
grib_api library is used:
module load grib_api/1.23.1-CrayGNU-18.08 cc -I$EBROOTGRIB_API/include -o mygribprogram mygribprogram.c -L$EBROOTGRIB_API/lib -lgrib_api
The EasyBuild software management system that NeSI/NIWA use to provide modules automatically defines environment variables
$EBROOT<library name in upper case> when a module is loaded, which help pointing the compiler and linker to include files and libraries as in the example above. If you are unsure which
$EBROOT<...> variables are available, use
module show grib_api/1.23.1-CrayGNU-18.08
to find out.
Note that specifying search paths with
-L is not strictly necessary in case of the GNU and Intel compilers, which will use the contents of
LD_LIBRARY_PATH provided by the NeSI/NIWA module. This will not work with the Cray compiler.
Important note: Make sure that you load the correct variant of a library, depending on your choice of compiler. Switching compiler environment will not switch NeSI/NIWA modules automatically. Furthermore, loading a NeSI/NIWA module may switch programming environment if it was built with a different compiler.
As mentioned earlier, EasyBuild uses the following module naming conventions (“toolchain names”) to identify the programming environment that was used to build the software:
CrayCCEfor libraries and tools built with the Cray compilers (
CrayIntelfor libraries and tools built with the Intel compilers (
CrayGNUfor libraries and tools built with the GNU compilers (
Using your own libraries
Linking against libraries that you built yourself is the same as linking against libraries provided by NeSI/NIWA - you will just need to point the compiler to the location where the include and library files are using the
Static and dynamic linking
The XC50 compilers drivers default to static linking where possible for maximum efficiency, avoiding the need to load shared libraries for hundreds or thousands of MPI ranks at runtime. If all dependencies are available as static libraries, the resulting executables will be completely self-contained (although they may still need the Cray MPI environment at runtime).
Here is an example that shows how to find out how your code was linked:
module load GSL/2.4-CrayGNU-2017.06 cc -I$EBROOTGRIB_API/include -o mygribprogram mygribprogram.c -L$EBROOTGRIB_API/lib -lgrib_api ldd mygribprogram
If you see the message
not a dynamic executable, your program was statically linked. Otherwise you will see a list of shared library dependencies that are needed at runtime.
If you have to link your code dynamically, either set
in your build environment (useful when using complex build systems), or add the
-dynamic flag to the compiler driver commands, e.g.,
cc -I$EBROOTGRIB_API/include -o mygribprogram mygribprogram.c -L$EBROOTGRIB_API/lib -lgrib_api -dynamic
ldd tool, you should now see a number of libraries that are dynamically linked.
You may occassionally see a warning message of the kind:
/opt/cray/pe/hdf5/22.214.171.124/INTEL/16.0/lib/libhdf5.a(H5PL.o): In function `H5PL_load': H5PL.c:(.text+0x612): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
This simply means that the library must be accessible at runtime despite fully static linking and the program is thus not entirely self-contained, which is usually not an issue.
Common linker problems
Linking can easily go wrong. Most often, you will see linker errors about “missing symbols” when the linker could not find a function used in your program or in one of the libraries that you linked against. To resolve this problem, have a closer look at the function names that the linker reported:
- Are you missing some object code files (these are compiled source files and have suffix
.o) that should appear on the linker line? This can happen if the build system was not configured correctly or has a bug. Try running the linking step manually with all source files and debug the build system (which can be a lengthy and cumbersome process, unfortunately).
- Do the missing functions have names that contain “mp” or “omp”? This could mean that some of your source files or external libraries were built with OpenMP support, which requires you to set an OpenMP flag (
-fopenmpfor GNU compilers,
-qopenmpfor Intel) in your linker command. For the Cray compilers, OpenMP is enabled by default and can be controlled using
- Do you see a very long list of complex-looking function names, and does your source code or external library dependency include C++ code? You may need to explicitly link against the C++ standard library (
-lstdc++for GNU and Cray compilers,
-cxxlibfor Intel compilers); this is a particularly common problem for statically linked code.
- Do the function names end with an underscore (“_”)? You might be missing some Fortran code, either from your own sources or from a library that was written in Fortran, or parts of your Fortran code were built with flags such as
-assume nounderscore(Intel) or
-fno-underscoring(GNU), while others were using different flags (note that the Cray compiler always uses underscores).
- Do the function names end with double underscores (“__”)? Fortran compilers offer an option to add double underscores to Fortran subroutine names for compatibility reasons (
-f[no-]second-underscore) which you may have to add or remove.
Note that the linker requires that function names match exactly, so any variation in function name in your code will lead to a “missing symbols” error (with the exception of character case in Fortran source code).
Building code on the CS500 platform
Building code on the CS500 platform is different from the XC50 platform:
- The CS500 platform does not currently use compiler drivers (these will be made available by Cray in the near future)
- The CS500 module environment can be reset using
module purgewithout problems - you will need to run
module load NeSIafterwards to make the NeSI software stack available again.
Building code on the CS500 platform follows the same process as building code on Mahuika. The only difference is that CS500 nodes use Intel Skylake CPUs, while Mahuika’s CS400 nodes use the older Intel Broadwell CPUs. This means that programs that were compiled on the CS500 platform may fail to run on Mahuika, producing either an error message (if built with the Intel compiler), or an “illegal instruction” error (if built with the Cray or GNU compilers).
Please refer to section Building code on Mahuika for further instructions.