HPCC results on an HPC (beowulf-style) cluster using CentOS 6.2

I have finally had a chance to carryout a few HPCC benchmark runs. As with the Linpack benchmarks I have compiled the HPCC benchmark with MKL and with ATLAS. 

Generally speaking MKL continues to perform better than ATLAS but clearly some of the benchmarks are hardly affected at all by the choice of library, which should not be surprising as some of the tests relate to latency and memory performance.

It is worth bearing in mind that this two node cluster does not have its own separate network switch and thus results will vary more than in a cluster with dedicated networking.

MKL	ATLAS
HPL_Tflops=0.0689043	HPL_Tflops=0.025042
StarDGEMM_Gflops=6.98082	StarDGEMM_Gflops=1.95127
SingleDGEMM_Gflops=8.489	SingleDGEMM_Gflops=2.0767
PTRANS_GBs=0.285695	PTRANS_GBs=0.282756
MPIRandomAccess_LCG_GUPs=0.0107743	MPIRandomAccess_LCG_GUPs=0.0106545
MPIRandomAccess_GUPs=0.00980471	MPIRandomAccess_GUPs=0.011238
StarRandomAccess_LCG_GUPs=0.00292826	StarRandomAccess_LCG_GUPs=0.00305111
SingleRandomAccess_LCG_GUPs=0.0202582	SingleRandomAccess_LCG_GUPs=0.0209147
StarRandomAccess_GUPs=0.00294884	StarRandomAccess_GUPs=0.0030755
SingleRandomAccess_GUPs=0.0222603	SingleRandomAccess_GUPs=0.0219117
StarSTREAM_Copy=0.424375	StarSTREAM_Copy=0.49285
StarSTREAM_Scale=0.445834	StarSTREAM_Scale=0.52021
StarSTREAM_Add=0.596884	StarSTREAM_Add=0.629714
StarSTREAM_Triad=0.640154	StarSTREAM_Triad=0.684154
SingleSTREAM_Copy=3.0544	SingleSTREAM_Copy=3.06379
SingleSTREAM_Scale=3.03374	SingleSTREAM_Scale=3.03051
SingleSTREAM_Add=3.339	SingleSTREAM_Add=3.332
SingleSTREAM_Triad=3.32887	SingleSTREAM_Triad=3.31208
StarFFT_Gflops=0.242265	StarFFT_Gflops=0.233275
SingleFFT_Gflops=0.894628	SingleFFT_Gflops=0.93583
MPIFFT_Gflops=0.55462	MPIFFT_Gflops=0.402139
MaxPingPongLatency_usec=717.918	MaxPingPongLatency_usec=499.696
RandomlyOrderedRingLatency_usec=393.943	RandomlyOrderedRingLatency_usec=201.655
MinPingPongBandwidth_GBytes=0.0386536	MinPingPongBandwidth_GBytes=0.0312215
NaturallyOrderedRingBandwidth_GBytes=0.0129542	NaturallyOrderedRingBandwidth_GBytes=0.012228
RandomlyOrderedRingBandwidth_GBytes=0.021249	RandomlyOrderedRingBandwidth_GBytes=0.016799
MinPingPongLatency_usec=0.384119	MinPingPongLatency_usec=0.357628
AvgPingPongLatency_usec=332.504	AvgPingPongLatency_usec=291.221
MaxPingPongBandwidth_GBytes=3.87644	MaxPingPongBandwidth_GBytes=3.36689
AvgPingPongBandwidth_GBytes=0.839076	AvgPingPongBandwidth_GBytes=0.479826
NaturallyOrderedRingLatency_usec=585.89	NaturallyOrderedRingLatency_usec=293.112

Run HPCC on an HPC (beowulf-style) cluster using CentOS 6.2

The HPC Challenge benchmark or HPCC, expands on HPL by adding six extra benchmarks to provide a more complete, yet still fairly synthetic, picture of a cluster's performance.

I managed to build HPCC without issues using the Atlas library, once I read the instructions, but struggled significantly with MKL until it was pointed out to me the existence of this tool that can be used to generate the appropriate link lines and compiler flags.

I shall concentrate on the install of HPCC using MKL as the results from running various HPL tests, see this post, showed that performance is significantly better than using the Atlas library. MKL is not free, but it's probably worth its price given the performance boost in HPL. We shall find out whether the same is true for HPCC.

1. Install the Intel MKL library, which can be downloaded from here. Then installation is fairly straight forward. Unpack the tarball, run install.sh and follow the instructions.
2. Download HPCC from here and extract it to ../hpcc-1.4.1
3. Create Make.intel file on ../hpcc-1.4.1./hpl . See relevant content below:  

# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
#
#MPdir        = /usr/bin/mpi
#MPinc        = -I$(MPdir)/include
#MPlib        = /usr/lib64/mpich2/lib/libmpich.a
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
#
LAdir        = /opt/intel/mkl/lib/intel64
LAinc        = /opt/intel/mkl/include
LAlib =  -Wl,--start-group  $(LAdir)/libmkl_cdft_core.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_sequential.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread -lm
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
#
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) -I$(LAinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
HPL_OPTS     = -DHPL_CALL_CBLAS
#
# ----------------------------------------------------------------------
#
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC           = /usr/bin/mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -DMKL_ILP64 -m64
#
# On some platforms,  it is necessary  to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER       = /usr/bin/mpicc
LINKFLAGS    = $(CCFLAGS)


4. From ../hpcc-1.4.1 run make arch=intel to build HPCC.

Before you can run hpcc, an input file needs to be created. The easiest is to rename the existing _hpccinf.txt and then edit it. If you are familiar with HPL, then you'll be familiar with this file:

mv _hpccinf.txt hpccinf.txt
 
You can now run hpcc with:

mpiexec.hydra -n 4 ./hpcc

Results can be found here.

Linpack (HPL) results on an HPC (beowulf-style) cluster using CentOS 6.2

In my previous post, I described how to install and run Linpack (HPL) on a two node HPC (Beowulf-style) cluster running CentOS 6.2, in this post I will discuss some of the results from various Linpack runs that I have conducted in the past few days.

The first thing to note is that the HPL.dat file that is available post install is simply useless to extract any kind of meaningful performance numbers, so the file needs to be edited, but how? There is an online tool that will generate an HPL.dat file and this is what I have been using to provide me with some guidance of what to use. I have changed the number of equations, to generate a nice graph.

The first two tests that I ran, were ran using the configuration described in my previous post, I then recompiled Linpack with MKL and re-run the tests, see figure 1 below for results.

Figure 1 - WR11C2R4 test for various problem sizes with fixed block size of 128.

The highest value for Atlas, is 25.13 GFlops, whereas the highest result for MKL is 60.03 GFlops, which means that using MKL more than doubles performance. I was expecting a good increase in performance with MKL but a more than doubling of performance is extremely impressive, it's a shame that MKL is not free, but in a real cluster it's probably worth the cost.

The tool suggests that it would be possible to run a test for a problem size of ~41000, however, it seems that performance tanks after a problem size of 30000 for Atlas. MKL shows better performance, but still performance does go down. Execution time for a problem size of 35000 was ~ 7000 seconds for Atlas, I did not try with MKL for such a large problem size. The reason is probably due to memory swapping as there is higher memory usage than expect, which is something that I will need to investigate. 

The second test I ran was intended to investigate the effect of block size. I fixed the problem size (N) and varied the block size (NB), see figure 2 below.

Figure 2 - Influence of Block Size on performance

The gains from increased block size appear to top out at a block size of 168 for a problem size of 20000 and 256 for a problem size of 25000. I did run with a block size of 268, but performance was actually reduced (60.1 GFlops). The netlib guidelines, recommend a block size of less than 256, so it shouldn't be surprising that a bigger block size yields worse performance. Block size is balancing act between data distribution and computational granularity.

It is interesting to note that the maximum performance (68.7 GFlops) was achieved for a problem size of 30000 and a block size of 192, although to be fair, the difference between a block size of 192 and 256 is only 4%.

Also interesting is how much the data varies for a problem size of 30000, all I can say is that the servers in the cluster don't have a separate network and thus performance is unlikely to ever be constant.

The efficiency of the cluster is actually only 46%, which is appalling, but given the various limitations in the system it's perhaps not that surprising.

In my next post, I discuss how to install HPCC, which is more comprehensive benchmark tool.