CyEnce: Using MPI


Parallelism on CyEnce is obtained by using MPI. All accounts are set up so that MPI uses the high performance Infiniband communication network. To use MPI :

* To compile Fortran 77, Fortran 90, and Fortran 95 MPI programs use 
     mpiifort. To compile C and C++ MPI programs use either 
     mpiicc or mpiicpc. 
* Use the CyEnce PBS script_writer to write
     a script to submit to the batch scheduler. Remember that there
     are 16 processors per node, so for a 32 processor job, you only
     need 2 nodes.
* In the script use "mpirun -np 32 ./a.out" to start 32 MPI processes.
     OpenMPI's mpirun is aware of the nodes that PBS assigns to it,
     so no one else will use the nodes on which your MPI job is running.
* Make sure that the executable (a.out in the example above) resides in one of 
     the following locations:
       /home/user     (where 'user' is your user name)
       /work/group    (where 'group' is your group name, issue 'groups' to find it out)
       /ptmp
     All these locations are mounted on each of the compute nodes.
     Don't place the executable in the local filesystem (/tmp) as each node has its 
     own /tmp . Files placed into /tmp on the front end node won't be available on 
     the compute nodes, so mpirun won't be able to start processes on compute nodes.
* One can use the storage on the disk drive on each of the compute nodes by reading 
     and writing to $TMPDIR.  This is temporary storage that can be used only during 
     the execution of your program. Only processors executing on a node have access 
     to this disk drive.  Since 16 processors share this same storage, you must 
     include the rank of the executing MPI processes when reading and writing files 
     to $TMPDIR. The size of $TMPDIR is about 3 TB.

* The -e and -o PBS files are not available until PBS job finishes, so you
     may want to use "mpirun -np 12 a.out >& output_files".  Then you can see
     the output from CyEnce while the job is running. Alternatively you can use 
     qpeek command:
       "qpeek    job#" shows STDOUT while job is running.
       "qpeek -e job#" shows STDERR while job is running.