Amazon EC2 Micro instance Linpack (HPL) performance

Amazon allows users to test their cloud for free for the first year, there are of course quite a few limitations, perhaps the most important of which is that you can only use the micro instance, which has 613 MB of RAM, I guess the hypervisor memory footprint must account for the ~150 MB that would make it up to 768 MB of memory usage.

At any rate, I wanted to find out how well these machines would perform on Linpack, so I created a very small, 4 node, cluster using my own instructions and set it to work.

All I can say is that the performance is appalling, the best test was 0.72 Gflops, see below.

It took over an hour to run a problem size of 15000, but unfortunately it crashed before it finished, even so I can't image the performance improving by much. These instances are unlikely to have even a single core all to themselves and network traffic was within Amazon's network rather than dedicated networks.

This is hardly surprising as Amazon has cluster specific instances available, still I'm pretty disappointed at the results.

I might try again with MKL to see if the performance improves somewhat.


================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4        1000   128     2     2               0.93              7.223e-01
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0064465 ...... PASSED
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4        5000   128     2     2             254.06              3.282e-01
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0061108 ...... PASSED
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR11C2R4       10000   128     2     2             965.92              6.903e-01
--------------------------------------------------------------------------------
||Ax-b||_oo/(eps*(||A||_oo*||x||_oo+||b||_oo)*N)=        0.0060791 ...... PASSED
================================================================================

Finished      3 tests with the following results:
              3 tests completed and passed residual checks,
              0 tests completed and failed residual checks,
              0 tests skipped because of illegal input values.
--------------------------------------------------------------------------------

Run HPCC on an HPC (beowulf-style) cluster using CentOS 6.2

The HPC Challenge benchmark or HPCC, expands on HPL by adding six extra benchmarks to provide a more complete, yet still fairly synthetic, picture of a cluster's performance.

I managed to build HPCC without issues using the Atlas library, once I read the instructions, but struggled significantly with MKL until it was pointed out to me the existence of this tool that can be used to generate the appropriate link lines and compiler flags.

I shall concentrate on the install of HPCC using MKL as the results from running various HPL tests, see this post, showed that performance is significantly better than using the Atlas library. MKL is not free, but it's probably worth its price given the performance boost in HPL. We shall find out whether the same is true for HPCC.

1. Install the Intel MKL library, which can be downloaded from here. Then installation is fairly straight forward. Unpack the tarball, run install.sh and follow the instructions.
2. Download HPCC from here and extract it to ../hpcc-1.4.1
3. Create Make.intel file on ../hpcc-1.4.1./hpl . See relevant content below:  

# ----------------------------------------------------------------------
# - HPL Directory Structure / HPL library ------------------------------
# ----------------------------------------------------------------------
#
TOPdir       = ../../..
INCdir       = $(TOPdir)/include
BINdir       = $(TOPdir)/bin/$(ARCH)
LIBdir       = $(TOPdir)/lib/$(ARCH)
#
HPLlib       = $(LIBdir)/libhpl.a
#
# ----------------------------------------------------------------------
# - Message Passing library (MPI) --------------------------------------
# ----------------------------------------------------------------------
#
#MPdir        = /usr/bin/mpi
#MPinc        = -I$(MPdir)/include
#MPlib        = /usr/lib64/mpich2/lib/libmpich.a
#
# ----------------------------------------------------------------------
# - Linear Algebra library (BLAS or VSIPL) -----------------------------
# ----------------------------------------------------------------------
#
LAdir        = /opt/intel/mkl/lib/intel64
LAinc        = /opt/intel/mkl/include
LAlib =  -Wl,--start-group  $(LAdir)/libmkl_cdft_core.a $(LAdir)/libmkl_intel_lp64.a $(LAdir)/libmkl_sequential.a $(LAdir)/libmkl_core.a $(LAdir)/libmkl_blacs_intelmpi_lp64.a -Wl,--end-group -lpthread -lm
# ----------------------------------------------------------------------
# - F77 / C interface --------------------------------------------------
# ----------------------------------------------------------------------
#
F2CDEFS      =
#
# ----------------------------------------------------------------------
# - HPL includes / libraries / specifics -------------------------------
# ----------------------------------------------------------------------
#
HPL_INCLUDES = -I$(INCdir) -I$(INCdir)/$(ARCH) -I$(LAinc)
HPL_LIBS     = $(HPLlib) $(LAlib) $(MPlib)
#
# - Compile time options -----------------------------------------------
#
HPL_OPTS     = -DHPL_CALL_CBLAS
#
# ----------------------------------------------------------------------
#
HPL_DEFS     = $(F2CDEFS) $(HPL_OPTS) $(HPL_INCLUDES)
#
# ----------------------------------------------------------------------
# - Compilers / linkers - Optimization flags ---------------------------
# ----------------------------------------------------------------------
#
CC           = /usr/bin/mpicc
CCNOOPT      = $(HPL_DEFS)
CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -DMKL_ILP64 -m64
#
# On some platforms,  it is necessary  to use the Fortran linker to find
# the Fortran internals used in the BLAS library.
#
LINKER       = /usr/bin/mpicc
LINKFLAGS    = $(CCFLAGS)


4. From ../hpcc-1.4.1 run make arch=intel to build HPCC.

Before you can run hpcc, an input file needs to be created. The easiest is to rename the existing _hpccinf.txt and then edit it. If you are familiar with HPL, then you'll be familiar with this file:

mv _hpccinf.txt hpccinf.txt
 
You can now run hpcc with:

mpiexec.hydra -n 4 ./hpcc

Results can be found here.

Linpack (HPL) results on an HPC (beowulf-style) cluster using CentOS 6.2

In my previous post, I described how to install and run Linpack (HPL) on a two node HPC (Beowulf-style) cluster running CentOS 6.2, in this post I will discuss some of the results from various Linpack runs that I have conducted in the past few days.

The first thing to note is that the HPL.dat file that is available post install is simply useless to extract any kind of meaningful performance numbers, so the file needs to be edited, but how? There is an online tool that will generate an HPL.dat file and this is what I have been using to provide me with some guidance of what to use. I have changed the number of equations, to generate a nice graph.

The first two tests that I ran, were ran using the configuration described in my previous post, I then recompiled Linpack with MKL and re-run the tests, see figure 1 below for results.

Figure 1 - WR11C2R4 test for various problem sizes with fixed block size of 128.

The highest value for Atlas, is 25.13 GFlops, whereas the highest result for MKL is 60.03 GFlops, which means that using MKL more than doubles performance. I was expecting a good increase in performance with MKL but a more than doubling of performance is extremely impressive, it's a shame that MKL is not free, but in a real cluster it's probably worth the cost.

The tool suggests that it would be possible to run a test for a problem size of ~41000, however, it seems that performance tanks after a problem size of 30000 for Atlas. MKL shows better performance, but still performance does go down. Execution time for a problem size of 35000 was ~ 7000 seconds for Atlas, I did not try with MKL for such a large problem size. The reason is probably due to memory swapping as there is higher memory usage than expect, which is something that I will need to investigate. 

The second test I ran was intended to investigate the effect of block size. I fixed the problem size (N) and varied the block size (NB), see figure 2 below.

Figure 2 - Influence of Block Size on performance

The gains from increased block size appear to top out at a block size of 168 for a problem size of 20000 and 256 for a problem size of 25000. I did run with a block size of 268, but performance was actually reduced (60.1 GFlops). The netlib guidelines, recommend a block size of less than 256, so it shouldn't be surprising that a bigger block size yields worse performance. Block size is balancing act between data distribution and computational granularity.

It is interesting to note that the maximum performance (68.7 GFlops) was achieved for a problem size of 30000 and a block size of 192, although to be fair, the difference between a block size of 192 and 256 is only 4%.

Also interesting is how much the data varies for a problem size of 30000, all I can say is that the servers in the cluster don't have a separate network and thus performance is unlikely to ever be constant.

The efficiency of the cluster is actually only 46%, which is appalling, but given the various limitations in the system it's perhaps not that surprising.

In my next post, I discuss how to install HPCC, which is more comprehensive benchmark tool.

Run Linpack (HPL) on an HPC (beowulf-style) cluster using CentOS 6.2

A few weeks ago I attended a symposium on HPC and Open Source and ever since I've been wanting to set up my own HPC cluster. So I did, here are the instructions to set up an HPC cluster using CentOS 6.2. 

I have set up a two node cluster, but these instructions could be used for any number of nodes. The servers I've used only have a single 74 GB hard drive, a single NIC, 8 GB of RAM and 2 quad core CPUs, so that the cluster has 16 cores and 16 GB of RAM.

1. Install CentOS using a minimum install to ensure that the smallest amount of packages get installed.
2. Enable NIC by editing NIC config file (/etc/sysconfig/network-scripts/ifcfg-eth0) (I used the text install and it seems to leave the NIC disabled, but it's quicker to navigate from the ILO interface):

DEVICE="eth0"
ONBOOT="yes"
BOOTPROTO=dhcp

3. Disable and stop the firewall (I'm assuming no internet access for your cluster, of course):

   chkconfig iptables off; service iptables stop

4. Install ssh clients and man. This installs the ssh client and scp among others things as well as man, which is always handy to have:
   

   yum -y install openssh-clients man

5. Modify ssh client configuration to allow seamless addition of hosts to the cluster. Add this line to /etc/ssh/ssh_config (Note that this is a security risk if your cluster has access to the internet):

   StrictHostKeyChecking no

6. Generate pass-phrase free key. This will make it easier to add hosts to the cluster (just press enter repeatedly after running ssh-keygen):
   

   ssh-keygen

7. Install compilers and libraries (Note that development packages were obtained from here and yum was run from the directory containing them):
   

   yum -y install gcc gcc-c++ atlas blas lapack  mpich2 make mpich2-devel atlas-devel
   

8. Add node hostname to /etc/hosts.
9. Create file /$(HOME)/hosts and add node hostname to it.

This creates a single node and thus it would be a bit of a stretch to call it a cluster, but adding extra nodes is as simple as repeating steps 1-9.  A few extra steps are needed, though, to ensure smooth running:

1. Add each extra node to the hosts file (/etc/hosts) of all nodes [A DNS server could be set up instead.] and to (/$(HOME)/hosts).
2. Copy key generated in step 5 to all nodes (If you don't have a head node, i.e. a node that does not do any calculations, remember to add the key to itself too):

   ssh-copy-id hostname

I have not made any comments on networking and this is because the servers that I have been using only have a single NIC as mentioned above. There are gains to be made by forcing as much intra-node communication as possible through the loopback interface, but this requires unique (/etc/hosts) files for each node and my original plan was to set up a 16 node cluster.

SELinux does not seem to have any negative effects, so I have left it on. I plan to test without it to see whether performance is improved.

At this point all that remains is to add some software that can run on the cluster and there is nothing better than HPL or Linpack, which is widely used to measure cluster efficiency (the ratio between theoretical and actual performance). Do the following steps on all nodes:

1. Download HPL from netlib.org and extract it to your home directory.
2. Copy Make.Linux_PII_CBLAS file from  $(HOME)/hpl-2.0/setup/ to $(HOME)/hpl-2.0/
3. Edit Make.Linux_PII_CBLAS file (Changes in Bold. Note that the MPI section is commented out):

# ----------------------------------------------------------------------

# - HPL Directory Structure / HPL library ------------------------------

# ----------------------------------------------------------------------

#

TOPdir       = $(HOME)/hpl-2.0

INCdir       = $(TOPdir)/include

BINdir       = $(TOPdir)/bin/$(ARCH)

LIBdir       = $(TOPdir)/lib/$(ARCH)

#

HPLlib       = $(LIBdir)/libhpl.a

#

# ----------------------------------------------------------------------

# - Message Passing library (MPI) --------------------------------------

# ----------------------------------------------------------------------

# MPinc tells the  C  compiler where to find the Message Passing library

# header files,  MPlib  is defined  to be the name of  the library to be

# used. The variable MPdir is only used for defining MPinc and MPlib.

#

#MPdir        = /usr/lib64/mpich2

#MPinc        = -I$(MPdir)/include

#MPlib        = $(MPdir)/lib/libmpich.a

#

# ----------------------------------------------------------------------

# - Linear Algebra library (BLAS or VSIPL) -----------------------------

# ----------------------------------------------------------------------

# LAinc tells the  C  compiler where to find the Linear Algebra  library

# header files,  LAlib  is defined  to be the name of  the library to be

# used. The variable LAdir is only used for defining LAinc and LAlib.

#

LAdir        = /usr/lib64/atlas

LAinc        =

LAlib        = $(LAdir)/libcblas.a $(LAdir)/libatlas.a

# ----------------------------------------------------------------------

# - Compilers / linkers - Optimization flags ---------------------------

# ----------------------------------------------------------------------

#

CC           = /usr/bin/mpicc

CCNOOPT      = $(HPL_DEFS)

CCFLAGS      = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops

#

# On some platforms,  it is necessary  to use the Fortran linker to find

# the Fortran internals used in the BLAS library.

#

LINKER       = /usr/bin/mpicc

LINKFLAGS    = $(CCFLAGS)

#

ARCHIVER     = ar

ARFLAGS      = r

RANLIB       = echo

#

# ----------------------------------------------------------------------

4. Run make arch=Linux_PII_CBLAS.  
5. You can now run Linpack (on a single node):

    cd bin/Linux_PII_CBLAS

   mpiexec.hydra -n 4 ./xhpl 

Repeat steps 1- 5 on all nodes and the you can now run Linpack on all nodes like this (from directory $(HOME)/hpl-2.0/Linux_PII_CBLAS/ ):

mpiexec.hydra -f /$(HOME)/hosts -n x ./xhpl 

where x is the number of cores in your cluster.

For results of running Linpack, see my next post here.