General usage

The Portland Group (PGI) compiler suite includes Fortran 90/95, Fortran 77, C and C++ compilers. These are the default compilers on Louhi and  the PGI programming environment module PrgEnv-pgi is loaded when logging in. The default version of PGI on Louhi is 9.0.2, but also older versions are available.

Compiler commands

The Cray compiler wrapper scripts (ftn, f77, cc, CC) should always be used for compilation on Louhi. The wrappers automatically link MPI, SciLib and a few other libraries, if needed. ftn is used for compiling and linking Fortran 9X programs, f77 for Fortran 77 programs, cc for C-programs and CC for C++-programs. Don't use directly the PGI compiler commands (pgf77, pgf90, pgf95, pgfortran, pgcc, pgCC, pgcpp) the linker ld command or the wrapper scripts (linux-77,linux-f90, linux-cc and linux-CC) used internally by the compiler wrapper scripts, because the resulting executables will not run on the compute nodes of Louhi.

A wrapper performs all or some of the following steps:

1. Preprocess the source text file
2. Check the syntax of the source code
3. Generate assembly language file
4. Pass control to subsequent assembly and linking steps
5. Create the executable

Compiler options

The Cray drivers accepts almost all normal compiler and linker options that PGI compilers support.  Most of the compiler and linker options are same for all PGI compilers.  There are, however some special options for Fortran, C or C++ which are discussed in separate sections in this document.The following table lists some of the general PGI compiler options:

Option	Description
-tp barcelona-64	AMD Barcelona processor in 64-bit mode. Added automatically by the module xtpe-barcelona.
-tp shanghai-64	AMD Barcelona processor in 64-bit mode. Added automatically by the module xtpe-shanghai.
-F	Preprocess files and save output in filename.f
-fast	Macro for a common set of optimizations, including SSE/SSE2 instructions. Recommended.
-O{0 | 1 | 2 | 3}	Optimization level (none to max, default 2)
-Mipa	Enable and specify options for interprocedural analysis
-Mcache_align	Align data along cache line boundaries
-c	Compile only, produce unlinked object filename.o
-g	Generate symbolic debug information
-gopt	Generate debug information, keeping optimizations on
-pg	Instrument program for profiling
-Minfo	Generate an optimization report to standard error
-Mneginfo	Information about why certain optimizations are not performed
-Mlist	Generate a listing file in filename.lst
-help -option	Give help about option -option
-mp	Generate multi-threaded code from OpenMP directives
-Mbounds	Turn on array bound checking

By default, the compilers produce 64-bit code. The compute nodes also have only 64-bit libraries. For the AMD quad-core Barcelona Opteron architecture there is the option -tp barcelona-64, which is added automatically. It is added by loading the module xtpe-barcelona, which should have been loaded automatically when you log in Louhi. This option is available version 7 or later PGI compilers on Louhi. From the version 8 onwards there has been the corresponding option -tp shanghai-64 for the AMD quad-core Shanghai Opteron architecture. It is added automatically if the module xtpe-shanghai is loaded instead of  xtpe-barcelona.

Version 6 PGI compilers do not contain this option and therefore, if a version 6 compiler must be used, the option must be removed before compilations from the environment variable PGI_POST_COMPILE_OPTS (see Compilers) redefining it, e.g., by the following way (a tsch example):

PGI_POST_COMPILE_OPTS="-tp amd64e"

But when you do this, you do not get full perfomance from quad-core processors.

The option -fast is generally strongly recommended to be used. This is an aggregate option that includes a number of individual PGI compiler options (-Mvect=sse -Mscalarsse -Mcache_align -Mflushz).

The option -Mipa that enables interprocedural analysis (IPA) for optimization has several suboptions. The options -Mipa=fast or -Mipa=fast,inline are often good choices. The option can be used when compiling in a single step, several steps or via the Make system. IPA produces special object files which may be several times larger than normal ones. See the man pages for more information on how to use the interprocedural analysis.

The option -Mcache_align aligns unconstrained objects on cache-line boundaries, where possible.  This doesn't affect a member of an aggregate structure or common block nor an allocatable or automatic arrays. To have an influence on the alignment of stack-based local variables, the main program or function must also be compiled with -Mcache_align. Note that -fastsse includes -Mcache_align.

 -Minfo, -Mneginfo and  -dryrun are useful if you need information about compilation and optimization.

The option -help gives information about compiler options when used with the PGI compiler commands (pgf77, pgf90, pgf95, pgfortran, pgcc, pgCC, pgcpp) :

pgcc -help -fastsse

Although the PGI compilers and the login nodes support creation of shared object files, the CNL OS on the compute nodes does not support using them. Therefore, you should always link everything statically on Louhi.

Recommended options

For both Fortran and C/C++ programs a good set of options is:

cc -fast cprog.c

or even

cc -fast -Mipa=fast,inline cprog.c

Optimization level is -O2 here. You may try also to add -O3, which is the highest optimization level.

Pragmas (C/C++) and directives (Fortran)

Pragmas and directives inserted in program source code allow you to alter the
effects of certain command-line options and control various aspects of the compilation process for a specific routine or a specific program loop. Chapters 6 & 7 in the PGI user's guide cover the available pragmas and directives.

Fortran

The file suffix rules relevant to PGI Fortran are given by the following table:

Suffix	How compiled
.f90, .f95	Fortran 90/95, free form
.F90, .F95	As above, and preprocessed
.f	Fortran, fixed form
.F	As above, and preprocessed
.hpf	High Performance Fortran (HPF)
.s	Assembly language
.o	Object file
.a	Library of object files

Compiler options

The following table lists some of the PGI Fortran specific compiler options:
 

Option	Description
-M {fixed | free}	Fixed/free form source code, override file suffix rule
-Mpreprocess	Preprocess code, override file suffix rule
-Mrecursive	Allow recursive procedures
-i8	Treat INTEGER variables as 8 bytes
-r8	Interpret REAL variables as 8 bytes

C/C++

PGI (cc) supports ANSI C99 (ANSI/ISO 9899-1999) , ANSI C (ANSI/ISO 9899-1990) and K&R C.  The options -c9x and -c99 are for C99 code.  PGI (CC) also supports ANSI C++ with cfront features.

Please note, that there is a name conflict between stdio.h and the C++ binding in the MPI include file mpi.h. This can be avoided by including mpi.h before stdio.h or the iostream routine.

The file suffix rules relevant to PGI C and C++ are given by the following table:

Suffix	How compiled
.c	C and C++ source, preproces and compile
.C	C++ source, preproces and compile
.cc	C++ source, preproces and compile
.cpp	C++ source, preproces and compile
.s	Assembler source, assemble
.S	Assember sorce, preprocess and assemble
.o	Object file, passed to linker
.a	Library of object files, passed to linker

Compiler options

For C++ programs add -minline=levels:10 --no_exceptions:

CC -fastsse -Mipa=fast -Minline=levels:10 --no_exceptions Cprog.C

However, if exception handling is used, a program compilation  with            --no_exceptions will fail. Inlining may boost performance of C programs, too. Please, note that PGI C and C++ compilers are not able to inline functions which are not in the same source file which you are compiling. If you want this inlining to happen, you must write  or move the functions to be inlined to the same file, which you are compiling.

Unsupported C++ Header Files

PGI does not provide a complete set of the old  deprecated C++ Standard Library and
STL header files. PGI C++ does support some old header files (iostream.h,
exception.h, iomanip.h, ios.h, istream.h, ostream.h, new.h,
streambuf.h, strstream.h, and typeinfo.h), which include their C++
Standard Library counterpart.

To use an unsupported header file, delete the .h. For example, change <vector.h> to <vector>, or create your own vector.h file and use the -I compiler option to cause the compiler to access the header file in your directory:

#ifndef __VECTOR_H
#define __VECTOR_H
#include <vector>
using std::vector;
#endif

Further information

The PGI on-line documentation offers all manuals in pdf format.

See the  man ftn and man f77 for a discussion of the Cray wrappers.

For more detailed information on the PGI compiler options for Fortran, see the man pages for pgf77 or pgf95, or the section Optimization.

The options -g, -pg and others needed for using the debugger and profiling tools are discussed in detail in the section Program development tools.

Louhi is intended to be used for running parallel codes only. For further information on compiling parallel codes on Louhi, see the sections Using MPI and Shared memory parallelization.

Louhi has currently 10 licence seats for the PGI compilers. A single licence is reserved for the user for the duration of the compilation.