TAO is successfully installed with MPI support on tembusu2, but failed to do it on angsana, though. It is found that Petsc compiles with the mpi installed on /opt/mpich/ for tembusu2. For angsana, Petsc should compile with /opt/mpich-64/, except for some glitches: perhaps this version of mpich was not compiled with the option -enable-sharedlib, and Petsc requires that mpich be compiled with this option.
No guarantees for this part, since in the end, the mpi already installed on tembusu2 is used, instead of installing MPICH. After unzipping mpich, just do a configure and a make as follows:
./configure --prefix=/usr/local/mpich-1.2.6 --enable-sharedlib
make
On angsana, you can try the following, and you will find that things doesn't work.
PETSC_ARCH=linux-gnu
PETSC_DIR=`pwd`
export PETSC_ARCH
PETSC_DIR
./config/configure.py# the configure process will find
mpich
./config/configure.py --with-mpi-dir=/opt/mpich-64/ # the configure
process will find mpich
make BOPT=O_c++
On tembusu2, on the other hand, things work out fine!
PETSC_ARCH=linux
PETSC_DIR=`pwd`
export
PETSC_ARCH PETSC_DIR
Do this, and the configure process will find mpich
./config/configure.py --with-shared
--with-clanguage=C++
or
./config/configure.py --with-mpi-dir=/opt/mpich/ --with-shared --with-clanguage=C++
then
make BOPT=O_c++
or do
make BOPT=g_c++
If you compile with O_c++ (or g_c++) here, make sure you do the same when you compile TAO. g_c++ will show you more debugging information when your programs run. Make sure that switches --with-shared --with-clanguage=C++ are included because these are not default settings. From PETSc 2.3.0, shared libraries are turned off by default. You should see MUCH smaller executables than not using shared libraries. --with-clanguage=C++ is required by TAO 1.8, but not by the earlier releases.
TAO_DIR=/home/username/tao-1.8
export TAO_DIR
make BOPT=O_c++ all
or
make BOPT=g_c++ all
In case some libraries are not found while compiling TAO, make sure they are in the petsc library directory. If not, recompile Petsc with the correction options.
TAO_DIR=/home/userID/tao-1.8
PETSC_DIR=/home/userID/petsc-2.3.0/
export
TAO_DIR PETSC_DIR
In order to run an mpich program with TAO, you have to create a file of nodes usable for mpi. See
https://www.comp.nus.edu.sg/cf/tembusu/mpi.html
for an explanation. In order to run an mpi program, you have to do something like
/opt/mpich/bin/mpirun -machinefile ~/machine-file -np 8 yourprogram yourprogramparameters
where an example of a machine-file (the one I am using now) for tembusu2 is
# cat
mynodes
access0
access1
access2
access3
access4
access5
access6
access7
access8
access9
access10
access11
access12
For a simple example of an mpi program, see
http://www.pdc.kth.se/training/Tutor/MPI/Basics.Lab/karp.soln.c
Acknowledgements
Many thanks to Mr Hai Leong Chieu for providing a lot of technical supports.
-----------------------------------------
The tree which fills the arms grew from the tiniest sprout; the tower of nine storeys rose from a (small) heap of earth; the journey of a thousand li commenced with a single step.
-- Lao Tze