=============================================================
Performance Tuning and setting up the input data file HPL.dat

Current as of release HPL - 2.1 - October 26, 2012
Updated with HPL-NVIDIA requirements
=============================================================
Check out  the website  www.netlib.org/benchmark/hpl  for the
latest information.

You may want to modify the input data file  HPL.dat.  Several
sample files are provided for convenience.  Pass  the
file name as a parameter to the executable  xhpl.  This  file
contains  information about the problem sizes, machine confi-
guration, and algorithm features to be used by the executable.
It is 30 lines long. All  the  selected  parameters  will  be
printed  in the output generated by the executable.

At the end of this file,  there are a couple of  experimental
guidelines that you may find useful.

=============================================================
File HPL.dat (description):

Line 1: (unused) Typically  one  would  use this line for its
own good. For example, it could be used to summarize the con-
tent of the input file. By default this line reads:

HPL Linpack benchmark input file

Line 2: (unused) same as line 1. By default this line reads:

Innovative Computing Laboratory, University of Tennessee

Line 3: the  user  can  choose where the output should be re-
directed to.  In the case of a file, a name is necessary, and
this  is  the  line  where one wants to specify it.  Only the
first name on this line is significant. By default, the line reads:

HPL.out  output file name (if any)

This  means  that if the user chooses to redirect the output to a
file, the file will be called "HPL.out". The rest of the line
is unused,  and this space to put some informative comment on
the meaning of this line.

Line 4: This line specifies  where the  output should go. The
line is formatted, it must be a positive integer, and the rest is
insignificant. Three choices are possible  for  the  positive
integer: 6 means that the output will go the standard output,
7 means that the  output will go to standard  error.  Any
other  integer  means  that  the  output should be redirected
to a file,  which  name has been specified in the line above.
This line by default reads:

6        device out (6=stdout,7=stderr,file)

which  means  that  the  output generated  by  the executable
should be redirected to the standard output.

Line 5: This line specifies the number of problem sizes to be
executed. This number should be less than or equal to 20. The
first  integer  is  significant,  the rest is ignored. If the
line reads:

3        # of problems sizes (N)

this  means  that  the user is willing to run 3 problem sizes
that will be specified in the next line.

Line 6:  This  line  specifies the problem sizes one wants to
run.  Assuming  the  line  above  started with 3, the 3 first
positive  integers  are significant, the rest is ignored. For
example:

3000 6000 10000    Ns

means that one wants xhpl to run 3 (specified in line 5) pro-
blem sizes, namely 3000, 6000 and 10000.

Line 7: This line specifies the number of block sizes  to  be
run. This number should be less than  or  equal  to  20.  The
first integer is significant, the rest  is  ignored.  If  the
line reads:

5        # of NBs

this means that the user is willing to use 5 block sizes that
will be specified in the next line.

Line 8: This line specifies the block sizes one wants to run.
Assuming  the line above started with 5, the 5 first positive
integers are significant, the rest is ignored. For example:

80 100 120 140 160 NBs

means  that  one  wants  xhpl  to use 5 (specified in line 7)
block sizes, namely 80, 100, 120, 140 and 160.

For best results: NB should be a multiple of 64.

Line 9 specifies how the  MPI processes should be mapped onto
the nodes of your platform.  There are currently two possible
mappings, namely row- and column-major. This feature is main-
ly  useful  when these nodes  are  themselves multi-processor
computers. A row-major mapping is recommended.

Line 10: This line specifies  the  number  of process grid to
be  run.  This  number  should  be  less than or equal to 20.
The first integer is significant, the rest is ignored. If the
line reads:

2        # of process grids (P x Q)

this  means  that you are willing to try 2 process grid sizes
that will be specified in the next line.

Line 11-12: These  two  lines specify  the  number of process
rows  and  columns of each grid you want to run on.  Assuming
the line above (10) started with 2,  the 2 first positive in-
tegers of those two lines are significant,  the rest is igno-
red. For example:

1 2          Ps
6 8          Qs

means  that one wants to run  xhpl  on  2 process grids (line
10), namely 1 by 6 and 2 by 8.  Note:  In this example, it is
required then to start xhpl on at least 16 nodes  (max of P_i
xQ_i). The runs on the two grids will be consecutive.  If one
was starting xhpl on more than 16 nodes, say 52, only 6 would
be used for the first grid  (1x6) and then 16  (2x8) would be
used for the second grid.  The fact  that you started the MPI
job on 52 nodes,  will not make HPL use all of them.  In this
example, only 16 would be used. If one wants to run xhpl with
52 processes one needs to specify a grid of 52 processes, for
example the following lines would do the job:

4  2         Ps
13 8         Qs

Line 13: This  line  specifies  the  threshold  the residuals
should be compared to.  The  residuals  should be of order 1,
but are in practice slightly less than this, typically 0.001.
This  line  is  made of a real number, the rest is insignifi-
cant. For example:

16.0         threshold

In practice,  a value of 16.0 will cover most cases.  For va-
rious reasons,  it is possible that some of the residuals be-
come slightly larger, say for example 35.6.  xhpl  will  mark
those runs as failed,  however they can be considered as cor-
rect.  A run can be considered as failed if the residual is a
few order of magnitude  bigger than 1 for example 10^6 or mo-
re. Note: if one was to specify a threshold of 0.0, all tests
would be marked  as failed, even though the answer is  likely
to be correct.  It is allowed to specify a negative value for
this threshold,  in  which case the checks will be by-passed,
no matter what the value is, as soon as it is negative.  This
feature allows saving time when performing a lot of experime-
nts, say for instance during the tuning phase. Example:

-16.0        threshold

The remaining lines  allow  specifying  algorithmic features.
xhpl  will  run  all  possible combinations of those for each
problem size, block size, and process grid combination.  This
is handy when one looks for an "optimal" set  of  parameters.
To understand a little bit better, lets say first a few words
about  the algorithm implemented in HPL. Basically this is  a
right-looking  version  with  row-partial pivoting. The panel
factorization is matrix-matrix operation based and recursive,
dividing the panel into  NDIV  subpanels  at each step.  This
part  of  the  panel   factorization   is  denoted  below  by
"recursive panel fact. (RFACT)". The recursion stops when the
current panel is made of less than or equal to NBMIN columns.
At  that  point,  xhpl  uses  a matrix-vector operation based
factorization denoted below by  "PFACTs".  Classic  recursion
would then use  NDIV=2,  NBMIN=1.  There  are  essentially  3
numerically  equivalent  LU  factorization algorithm variants
(left-looking, Crout  and  right-looking).  In  HPL,  one can
choose  every one  of those  for the  RFACT,  as well as  the
PFACT. The following lines of HPL.dat allows you to set those
parameters.

Lines 14-21: (Example 1)
3       # of panel fact
0 1 2   PFACTs (0=left, 1=Crout, 2=Right)
4       # of recursive stopping criterium
1 2 4 8 NBMINs (>= 1)
3       # of panels in recursion
2 3 4   NDIVs
3       # of recursive panel fact.
0 1 2   RFACTs (0=left, 1=Crout, 2=Right)

This  example  would  try all variants of PFACT, 4 values for
NBMIN,  namely 1, 2, 4 and 8,  3 values for NDIV namely 2,  3
and 4, and all variants for RFACT.  Lines 14-21: (Example 1)

2       # of panel fact
2 0     PFACTs (0=left, 1=Crout, 2=Right)
2       # of recursive stopping criterium
4 8     NBMINs (>= 1)
1       # of panels in recursion
2       NDIVs
1       # of recursive panel fact.
2       RFACTs (0=left, 1=Crout, 2=Right)

This example would try  2 variants of PFACT namely right loo-
king and left looking, 2 values for NBMIN, namely 4 and 8,  1
value for NDIV namely 2, and one variant for RFACT.

HPL-NVIDIA ignores the panel factorization parameters.

In the  main loop of the algorithm,  the current panel of co-
lumn is broadcast in process rows  using  a virtual  ring to-
pology. HPL offers various choices, and one most  likely want
to use the increasing ring modified encoded as 1.  4  is also
a good choice. Lines 22-23: (Example 1):

1       # of broadcast
1       BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)

This will cause HPL  to broadcast the current panel using the
increasing ring modified topology. Lines 22-23: (Example 2):

2       # of broadcast
0 4     BCASTs (0=1rg,1=1rM,2=2rg,3=2rM,4=Lng,5=LnM)

This will cause  HPL to broadcast the current panel using the
increasing ring virtual topology and the long message algori-
Thm.

HPL-NVIDIA ignores broadcast parameters.

Lines 24-25  allow  to  specify  the look-ahead depth used by
HPL. A depth of 0 means that the next panel is factorized af-
ter the update by the current panel is completely finished. A
depth of 1 means that the next panel is factorized immediate-
ly after being updated.  The  update by the current  panel is
then finished.  A depth of k means that the k next panels are
factorized immediately after being updated. The update by the
current  panel is then finished. It turns out that a depth of
1  seems  to give the best results, but may need a large pro-
blem size  before one can see the performance gain. So use 1,
if you do not know better,  otherwise  you may want to try 0.
Look-ahead of depths 2  and larger will probably not give you
better results.  Lines 24-25: (Example 1):

1       # of lookahead depth
1       DEPTHs (>=0)

This will cause HPL to use a look-ahead of depth 1.
Lines 24-25: (Example 2):

2       # of lookahead depth
0 1     DEPTHs (>=0)

This will cause HPL to use a look-ahead of depths 0 and 1.

HPL-NVIDIA ignores the look-ahead value. Look-ahead is built-
in instead.

Lines 26-27  allow to specify  the swapping algorithm used by
HPL for all tests.  There  are  currently  two swapping algo-
rithms  available,  one  based  on "binary exchange"  and the
other one based on a  "spread-roll"  procedure  (also  called
"long" below. For large problem sizes, this last one is like-
ly to be more efficient. The user can also choose to mix both
variants, that is "binary-exchange"  for  a number of columns
less  than a threshold value, and then the  "spread-roll" al-
gorithm.  This threshold  value is then specified on Line 27.
Lines 26-27: (Example 1):

1       SWAP (0=bin-exch,1=long,2=mix)
60      swapping threshold

This will cause HPL to use the "long" or  "spread-roll" swap-
ping algorithm.  Note  that a threshold  is specified in that
example but not used by HPL. Lines 26-27: (Example 2):

2       SWAP (0=bin-exch,1=long,2=mix)
60      swapping threshold

This will cause HPL to use the "long" or  "spread-roll" swap-
ping  algorithm  as  soon as there is more than 60 columns in
the row panel.  Otherwise,  the  "binary-exchange"  algorithm
will be used instead.

HPL-NVIDIA ignores the SWAP and swap-threshold values.

Line 28 specifies whether the upper triangle of the panel  of
columns  should  be  stored  in  no-transposed  or transposed
form. Example:

1            L1 in (0=transposed,1=no-transposed) form

HPL-NVIDIA expects L1 in no-transposed form for now,  i.e  L1
should be 1.

Line 29 specifies whether the panel of rows U should be stored
in no-transposed or transposed form. Example:

0            U  in (0=transposed,1=no-transposed) form

HPL-NVIDIA expects U in transposed form for now, i.e U should
be 0.

Line 30 enables/disables the equilibration phase. This option
will not be used unless you selected 1 or 2 in Line 26. Ex:

1            Equilibration (0=no,1=yes)

HPL-NVIDIA ignores equilibration parameter.
Line 31 specifies the alignment  in  memory  for  the  memory
space allocated by HPL.  On  modern  machines,  one  probably
wants to use 4, 8 or 16. This may result in a tiny amount  of
memory wasted. Example:

8       memory alignment in double words (> 0)

HPL-NVIDIA ignores memory alignment.

=============================================================
Guidelines:

1) HPL-NVIDIA performs better with larger problem sizes.  The
limit is determined by the available GPU  devices  memory.  A
rough estimate of memory size required is (N)*(N)*8 bytes.

2) Figure  out  a  good  block  size  for  the  matrix-matrix
multiply routine. The best method is to try a few out. If you
happen  to know  the block size  used  by  the  matrix-matrix
multiply routine, a small multiple of that block size will do
fine.

HPL  uses the block size NB for the data distribution as well
as  for   the  computational   granularity.   From   a   data
distribution point of view,  the smaller the NB, the better the
load balance.  You  definitely  want  to stay away  from very
large values of NB.  From a computation point of view,  a too
small value of  NB may limit the computational performance by
a large factor because almost no data reuse will occur in the
highest level of the memory hierarchy. The number of messages
will also increase.  Efficient  matrix-multiply  routines are
often internally blocked.  Small multiples  of  this blocking
factor are likely to be good block sizes for HPL.  The bottom
line  is  that  "good"  block sizes  are almost always in the
[128..2048] interval. The best values depend on the computat-
ion/communication performance ratio of your system. To a much
lesser extent, the  problem  size  matters  as  well.  Say  for
example, you emperically found that 1024 was a good block size
with respect to performance.

3)  The process mapping  should  not matter  if  the nodes of
your platform are single processor computers.  If these nodes
are multi-processors, a col-major mapping is recommended.

4) HPL likes "square" or slightly flat process grids.  Unless
you  are using  a very small process grid, stay away from the
1-by-Q and P-by-1 process grids.

=============================================================

HPL-NVIDIA takes several runtime environment variables to im-
prove the performance on different platforms.

`HPL_CUSOLVER_MP_TESTS` enables performance test for standalone
kernels: gemm,  panel factorization, broadcast,  gather, etc.
    a) 1 - enable (default)
    b) 0 - disable

`HPL_CUSOLVER_MP_TESTS_GEMM_ITERS` Number of repeat GEMM calls
in tests. (default 128)

`HPL_CHUNK_SIZE_NBS` the number of panels to split global upda-
tes into chunks, to overlap gemm and global swap. Any number >= 1
(default 16)

`HPL_DIST_TRSM_FLAG` Perform the solve step (TRSM) in parallell,
rather than on only the ranks that own that part of the
matrix.
    a) 1 - enable (default)
    b) 0 - disable

`HPL_CTA_PER_FCT` Sets the number of CTAs (thread blocks) for
factorisation. Any number > 0. (default 16)

`WARMUP_END_PROG` enables a warm-up run before the main HPL call
Any number (default 0) perform  X% of the whole HPL in warm-up
run.

`HPL_P2P_AS_BCAST` specifies which implementation of p2p broadcast
will be used
    a) 0 - ncclBcast
    b) 1 - ncclSend/Recv (default)
    c) 2 - CUDA-aware MPI
    d) 3 - MPI
    e) 4 - NVSHMEM

`HPL_USE_NVSHMEM` - enables NVSHMEM for inter-gpu communications
    a) 1 - enable (default)
    b) 0 - disable

`HPL_NVSHMEM_SWAP` - enables NVSHMEM in swap. This feature can
be enabled with HPL_USE_NVSHMEM only.
    a) 0 - disable (default)
    b) 1 - enable

`HPL_FCT_COMM_POLICY` - specifies which implementation of global
pivoting will be used
    a) 0 - NVSHMEM based (default)
    b) 1 - Host MPI
`HPL_FCT_COMM_POLICY=0` supports only column-major process mapping

HPL-NVIDIA Out-of-core mode:

`HPL_OOC_MODE` - activates out-of-core mode and enables the new feature
(default=0)

`HPL_OOC_NUM_STREAMS` - sets the number of streams used for copying data
to and from the GPU. The default value should be sufficient (default=3).

`HPL_OOC_TILE_M`, row blocking factor (default=4096).

`HPL_OOC_TILE_N`, column blocking factor (default=4096).

`HPL_ALLOC_HUGEPAGES`, uses transparent hugepages for host-side allocations.
This is done through the `madvise` system call and requires
`/sys/kernel/mm/transparent_hugepage/enabled` to be set to `madvise` to have
an effect (default=0).
