Parallelism on lightning3 is obtained by using MPI. All accounts are set up so that MPI uses the high performance Infiniband communication network. To use MPI :
* To compile Fortran 77, Fortran 90, and Fortran 95 MPI programs use either mpif77, mpif90 or mpif95. To compile C and C++ MPI programs use either mpicc or mpiCC. * The default MPI is Open MPI 1.5.3 with GNU compilers. To use Open MPI 1.8.3 with Intel compilers set PATH to /shared/openmpi-1.8.3/intel/bin/:$PATH * use the lightning3 PBS script_writer to write a script to submit to the batch scheduler. Remember that there are 16 processors per node, so for a 32 processor job, you only need 2 nodes. * In the script use mpirun -np 32 ./a.out OpenMPI's mpirun is aware of the nodes that PBS assigns to it, so no one else will use the nodes on which your MPI job is running. * Make sure that the executable (a.out in the example above) resides in one of the following locations: /home/user (where 'user' is your user name) /work/group (where 'group' is your group name, issue 'groups' to find it out) /ptmp All these locations are mounted on each of the compute nodes. Don't place the executable in the local filesystem (/tmp) as each node has its own /tmp . Files placed into /tmp on the front end node won't be available on the compute nodes, so mpirun won't be able to start processes on compute nodes. * One can use the storage on the disk drive on each of the compute nodes by reading and writing to $TMPDIR. This is temporary storage that can be used only during the execution of your program. Only processors executing on a node have access to this disk drive. Since 16 processors share this same storage, you must include the rank of the executing MPI processes when reading and writing files to $TMPDIR. The size of $TMPDIR is about 3 TB. * The -e and -o PBS files are not available until PBS job finishes, so you may want to use 'mpirun -np 12 a.out >& output_file' . Then you can see the output from lightning3 while the job is running. Alternatevily you can use qpeek command: qpeek job# shows STDOUT while job is running. qpeek -e job# shows STDERR while job is running. * For convenience an mpirun command can be submitted to the batch queues using the command bmpirun rather than mpirun. E.g. bmpirun -np 8 ./a.out Restrictions: no more than 16 processes no more than 1 hour output does not appear until after the command is complete. The command runs immediately if enough nodes are free, otherwise, it waits in the queue. If you use CNTL-C to exit the command, you also need to qdel the associated job in the PBS queues.