HPCC results on an HPC (beowulf-style) cluster using CentOS 6.2

I have finally had a chance to carryout a few HPCC benchmark runs. As with the Linpack benchmarks I have compiled the HPCC benchmark with MKL and with ATLAS. 

Generally speaking MKL continues to perform better than ATLAS but clearly some of the benchmarks are hardly affected at all by the choice of library, which should not be surprising as some of the tests relate to latency and memory performance.

It is worth bearing in mind that this two node cluster does not have its own separate network switch and thus results will vary more than in a cluster with dedicated networking.

MKL	ATLAS
HPL_Tflops=0.0689043	HPL_Tflops=0.025042
StarDGEMM_Gflops=6.98082	StarDGEMM_Gflops=1.95127
SingleDGEMM_Gflops=8.489	SingleDGEMM_Gflops=2.0767
PTRANS_GBs=0.285695	PTRANS_GBs=0.282756
MPIRandomAccess_LCG_GUPs=0.0107743	MPIRandomAccess_LCG_GUPs=0.0106545
MPIRandomAccess_GUPs=0.00980471	MPIRandomAccess_GUPs=0.011238
StarRandomAccess_LCG_GUPs=0.00292826	StarRandomAccess_LCG_GUPs=0.00305111
SingleRandomAccess_LCG_GUPs=0.0202582	SingleRandomAccess_LCG_GUPs=0.0209147
StarRandomAccess_GUPs=0.00294884	StarRandomAccess_GUPs=0.0030755
SingleRandomAccess_GUPs=0.0222603	SingleRandomAccess_GUPs=0.0219117
StarSTREAM_Copy=0.424375	StarSTREAM_Copy=0.49285
StarSTREAM_Scale=0.445834	StarSTREAM_Scale=0.52021
StarSTREAM_Add=0.596884	StarSTREAM_Add=0.629714
StarSTREAM_Triad=0.640154	StarSTREAM_Triad=0.684154
SingleSTREAM_Copy=3.0544	SingleSTREAM_Copy=3.06379
SingleSTREAM_Scale=3.03374	SingleSTREAM_Scale=3.03051
SingleSTREAM_Add=3.339	SingleSTREAM_Add=3.332
SingleSTREAM_Triad=3.32887	SingleSTREAM_Triad=3.31208
StarFFT_Gflops=0.242265	StarFFT_Gflops=0.233275
SingleFFT_Gflops=0.894628	SingleFFT_Gflops=0.93583
MPIFFT_Gflops=0.55462	MPIFFT_Gflops=0.402139
MaxPingPongLatency_usec=717.918	MaxPingPongLatency_usec=499.696
RandomlyOrderedRingLatency_usec=393.943	RandomlyOrderedRingLatency_usec=201.655
MinPingPongBandwidth_GBytes=0.0386536	MinPingPongBandwidth_GBytes=0.0312215
NaturallyOrderedRingBandwidth_GBytes=0.0129542	NaturallyOrderedRingBandwidth_GBytes=0.012228
RandomlyOrderedRingBandwidth_GBytes=0.021249	RandomlyOrderedRingBandwidth_GBytes=0.016799
MinPingPongLatency_usec=0.384119	MinPingPongLatency_usec=0.357628
AvgPingPongLatency_usec=332.504	AvgPingPongLatency_usec=291.221
MaxPingPongBandwidth_GBytes=3.87644	MaxPingPongBandwidth_GBytes=3.36689
AvgPingPongBandwidth_GBytes=0.839076	AvgPingPongBandwidth_GBytes=0.479826
NaturallyOrderedRingLatency_usec=585.89	NaturallyOrderedRingLatency_usec=293.112

Run HPCC on an HPC (beowulf-style) cluster using CentOS 6.2

The HPC Challenge benchmark or HPCC, expands on HPL by adding six extra benchmarks to provide a more complete, yet still fairly synthetic, picture of a cluster's performance.

I managed to build HPCC without issues using the Atlas library, once I read the instructions, but struggled significantly with MKL until it was pointed out to me the existence of this tool that can be used to generate the appropriate link lines and compiler flags.

I shall concentrate on the install of HPCC using MKL as the results from running various HPL tests, see this post, showed that performance is significantly better than using the Atlas library. MKL is not free, but it's probably worth its price given the performance boost in HPL. We shall find out whether the same is true for HPCC.

1. Install the Intel MKL library, which can be downloaded from here. Then installation is fairly straight forward. Unpack the tarball, run install.sh and follow the instructions.
2. Download HPCC from here and extract it to ../hpcc-1.4.1
3. Create Make.intel file on ../hpcc-1.4.1./hpl . See relevant content below:  

# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
#
#MPdir        = /usr/bin/mpi
#MPinc        = -I$(MPdir)/include
#MPlib        = /usr/lib64/mpich2/lib/libmpich.a
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
#
LAdir        = /opt/intel/mkl/lib/intel64
LAinc        = /opt/intel/mkl/include
LAlib =  -Wl,--start-group  $(LAdir)/libmkl_cdft_core.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_sequential.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread -lm
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
#
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) -I$(LAinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
HPL_OPTS     = -DHPL_CALL_CBLAS
#
# ----------------------------------------------------------------------
#
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC           = /usr/bin/mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -DMKL_ILP64 -m64
#
# On some platforms,  it is necessary  to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER       = /usr/bin/mpicc
LINKFLAGS    = $(CCFLAGS)


4. From ../hpcc-1.4.1 run make arch=intel to build HPCC.

Before you can run hpcc, an input file needs to be created. The easiest is to rename the existing _hpccinf.txt and then edit it. If you are familiar with HPL, then you'll be familiar with this file:

mv _hpccinf.txt hpccinf.txt
 
You can now run hpcc with:

mpiexec.hydra -n 4 ./hpcc

Results can be found here.

Linpack (HPL) results on an HPC (beowulf-style) cluster using CentOS 6.2

In my previous post, I described how to install and run Linpack (HPL) on a two node HPC (Beowulf-style) cluster running CentOS 6.2, in this post I will discuss some of the results from various Linpack runs that I have conducted in the past few days.

The first thing to note is that the HPL.dat file that is available post install is simply useless to extract any kind of meaningful performance numbers, so the file needs to be edited, but how? There is an online tool that will generate an HPL.dat file and this is what I have been using to provide me with some guidance of what to use. I have changed the number of equations, to generate a nice graph.

The first two tests that I ran, were ran using the configuration described in my previous post, I then recompiled Linpack with MKL and re-run the tests, see figure 1 below for results.

Figure 1 - WR11C2R4 test for various problem sizes with fixed block size of 128.

The highest value for Atlas, is 25.13 GFlops, whereas the highest result for MKL is 60.03 GFlops, which means that using MKL more than doubles performance. I was expecting a good increase in performance with MKL but a more than doubling of performance is extremely impressive, it's a shame that MKL is not free, but in a real cluster it's probably worth the cost.

The tool suggests that it would be possible to run a test for a problem size of ~41000, however, it seems that performance tanks after a problem size of 30000 for Atlas. MKL shows better performance, but still performance does go down. Execution time for a problem size of 35000 was ~ 7000 seconds for Atlas, I did not try with MKL for such a large problem size. The reason is probably due to memory swapping as there is higher memory usage than expect, which is something that I will need to investigate. 

The second test I ran was intended to investigate the effect of block size. I fixed the problem size (N) and varied the block size (NB), see figure 2 below.

Figure 2 - Influence of Block Size on performance

The gains from increased block size appear to top out at a block size of 168 for a problem size of 20000 and 256 for a problem size of 25000. I did run with a block size of 268, but performance was actually reduced (60.1 GFlops). The netlib guidelines, recommend a block size of less than 256, so it shouldn't be surprising that a bigger block size yields worse performance. Block size is balancing act between data distribution and computational granularity.

It is interesting to note that the maximum performance (68.7 GFlops) was achieved for a problem size of 30000 and a block size of 192, although to be fair, the difference between a block size of 192 and 256 is only 4%.

Also interesting is how much the data varies for a problem size of 30000, all I can say is that the servers in the cluster don't have a separate network and thus performance is unlikely to ever be constant.

The efficiency of the cluster is actually only 46%, which is appalling, but given the various limitations in the system it's perhaps not that surprising.

In my next post, I discuss how to install HPCC, which is more comprehensive benchmark tool.

Run Linpack (HPL) on an HPC (beowulf-style) cluster using CentOS 6.2

A few weeks ago I attended a symposium on HPC and Open Source and ever since I've been wanting to set up my own HPC cluster. So I did, here are the instructions to set up an HPC cluster using CentOS 6.2. 

I have set up a two node cluster, but these instructions could be used for any number of nodes. The servers I've used only have a single 74 GB hard drive, a single NIC, 8 GB of RAM and 2 quad core CPUs, so that the cluster has 16 cores and 16 GB of RAM.

1. Install CentOS using a minimum install to ensure that the smallest amount of packages get installed.
2. Enable NIC by editing NIC config file (/etc/sysconfig/network-scripts/ifcfg-eth0) (I used the text install and it seems to leave the NIC disabled, but it's quicker to navigate from the ILO interface):

DEVICE="eth0"
ONBOOT="yes"
BOOTPROTO=dhcp

3. Disable and stop the firewall (I'm assuming no internet access for your cluster, of course):

   chkconfig iptables off; service iptables stop

4. Install ssh clients and man. This installs the ssh client and scp among others things as well as man, which is always handy to have:
   

   yum -y install openssh-clients man

5. Modify ssh client configuration to allow seamless addition of hosts to the cluster. Add this line to /etc/ssh/ssh_config (Note that this is a security risk if your cluster has access to the internet):

   StrictHostKeyChecking no

6. Generate pass-phrase free key. This will make it easier to add hosts to the cluster (just press enter repeatedly after running ssh-keygen):
   

   ssh-keygen

7. Install compilers and libraries (Note that development packages were obtained from here and yum was run from the directory containing them):
   

   yum -y install gcc gcc-c++ atlas blas lapack  mpich2 make mpich2-devel atlas-devel
   

8. Add node hostname to /etc/hosts.
9. Create file /$(HOME)/hosts and add node hostname to it.

This creates a single node and thus it would be a bit of a stretch to call it a cluster, but adding extra nodes is as simple as repeating steps 1-9.  A few extra steps are needed, though, to ensure smooth running:

1. Add each extra node to the hosts file (/etc/hosts) of all nodes [A DNS server could be set up instead.] and to (/$(HOME)/hosts).
2. Copy key generated in step 5 to all nodes (If you don't have a head node, i.e. a node that does not do any calculations, remember to add the key to itself too):

   ssh-copy-id hostname

I have not made any comments on networking and this is because the servers that I have been using only have a single NIC as mentioned above. There are gains to be made by forcing as much intra-node communication as possible through the loopback interface, but this requires unique (/etc/hosts) files for each node and my original plan was to set up a 16 node cluster.

SELinux does not seem to have any negative effects, so I have left it on. I plan to test without it to see whether performance is improved.

At this point all that remains is to add some software that can run on the cluster and there is nothing better than HPL or Linpack, which is widely used to measure cluster efficiency (the ratio between theoretical and actual performance). Do the following steps on all nodes:

1. Download HPL from netlib.org and extract it to your home directory.
2. Copy Make.Linux_PII_CBLAS file from  $(HOME)/hpl-2.0/setup/ to $(HOME)/hpl-2.0/
3. Edit Make.Linux_PII_CBLAS file (Changes in Bold. Note that the MPI section is commented out):

# ----------------------------------------------------------------------

# - HPL Directory Structure / HPL library ------------------------------

# ----------------------------------------------------------------------

#

TOPdir       = $(HOME)/hpl-2.0

INCdir       = $(TOPdir)/include

BINdir       = $(TOPdir)/bin/$(ARCH)

LIBdir       = $(TOPdir)/lib/$(ARCH)

#

HPLlib       = $(LIBdir)/libhpl.a

#

# ----------------------------------------------------------------------

# - Message Passing library (MPI) --------------------------------------

# ----------------------------------------------------------------------

# MPinc tells the  C  compiler where to find the Message Passing library

# header files,  MPlib  is defined  to be the name of  the library to be

# used. The variable MPdir is only used for defining MPinc and MPlib.

#

#MPdir        = /usr/lib64/mpich2

#MPinc        = -I$(MPdir)/include

#MPlib        = $(MPdir)/lib/libmpich.a

#

# ----------------------------------------------------------------------

# - Linear Algebra library (BLAS or VSIPL) -----------------------------

# ----------------------------------------------------------------------

# LAinc tells the  C  compiler where to find the Linear Algebra  library

# header files,  LAlib  is defined  to be the name of  the library to be

# used. The variable LAdir is only used for defining LAinc and LAlib.

#

LAdir        = /usr/lib64/atlas

LAinc        =

LAlib        = $(LAdir)/libcblas.a $(LAdir)/libatlas.a

# ----------------------------------------------------------------------

# - Compilers / linkers - Optimization flags ---------------------------

# ----------------------------------------------------------------------

#

CC           = /usr/bin/mpicc

CCNOOPT      = $(HPL_DEFS)

CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops

#

# On some platforms,  it is necessary  to use the Fortran linker to find

# the Fortran internals used in the BLAS library.

#

LINKER       = /usr/bin/mpicc

LINKFLAGS    = $(CCFLAGS)

#

ARCHIVER     = ar

ARFLAGS      = r

RANLIB       = echo

#

# ----------------------------------------------------------------------

4. Run make arch=Linux_PII_CBLAS.  
5. You can now run Linpack (on a single node):

    cd bin/Linux_PII_CBLAS

   mpiexec.hydra -n 4 ./xhpl 

Repeat steps 1- 5 on all nodes and the you can now run Linpack on all nodes like this (from directory $(HOME)/hpl-2.0/Linux_PII_CBLAS/ ):

mpiexec.hydra -f /$(HOME)/hosts -n x ./xhpl 

where x is the number of cores in your cluster.

For results of running Linpack, see my next post here.