CyStorm - Using MPI


Parallelism on CyStorm is obtained by using MPI. All accounts are set up so that MPI uses the high performance Infiniband communication network. To use MPI :

* To compile Fortran 77, Fortran 90, and Fortran 95 MPI programs use mpif90. 
     To compile C and C++ MPI programs use either mpicc or mpic++. 

* Use the CyStorm PBS script_writer to write a script to submit to the batch 
     scheduler. 
* In the script use "mpirun -np 8 ./a.out" to start 8 MPI processes.
* Make sure that the executable (a.out in the example above) resides in either 
     /home/user (where 'user' is your user name) or /ptmp . Both these locations 
     are mounted on each of the compute nodes.
     Don't place the executable in the local filesystem (/tmp) as each node has  
     its own /tmp . Files placed into /tmp on the front end node won't be 
     available on the compute nodes, so mpirun won't be able to start processes 
     on compute nodes.
* One can use the storage on the disk drive on each of the compute nodes by 
     reading and writing to $TMPDIR.  This is temporary storage that can be used 
     only during the execution of your program. Only processors executing on a 
     node have access to this disk drive.  Since 16 processors share this same 
     storage, you must include the rank of the executing MPI processes when 
     reading and writing files to $TMPDIR. The size of $TMPDIR is about 3 TB.

* The -e and -o PBS files are not available until PBS job finishes, so you
     may want to use "mpirun -np 12 a.out >& output_files".  Then you can see
     the output from CyStorm while the job is running. Alternatively you can 
     use qpeek command:
       "qpeek    job#" shows STDOUT while job is running.
       "qpeek -e job#" shows STDERR while job is running.