lightningsmp: Detailed Description

General Description

lightningsmp is an AMD Opteron cluster currently with 512 processor cores (32 nodes) with each node having 64 GBytes of memory (2048 GB total memory on the cluster). Nodes are interconnected with an Infiniband communication network for low latency/high bandwidth MPI communication and a Gigabit Ethernet switch for I/O, management, and PBS communication. Lightningsmp shares fileservers with Lightning for a combined 18.2 TB of RAID-6 storage are integrated into the cluster. The peak performance of each processor is 4.8 Gflops so 512 processors gives a peak performance of 2,458 gigaflops (2.458 teraflops). The cluster was purchased from Atipa, see http://www.atipa.com

 

Front End Node

        Single processor, 2.1 GHz Quad-Core AMD Opteron 2352 (Barcelona) Processor with 2 MB on-chip cache and 16 GB of ECC memory

        Supermicro A+ series motherboard and chassis with two 500 watt, hot swappable, redundant power supplies

        Two mirrored, 160 GB SATA, enterprise quality, Western Digital (WD1600SD) disk drives

        Users logon, compiler, and launch batch jobs to compute nodes from the Front End Node.

 

32 Compute Nodes with each node consisting of the following:

        quad processor, 2.4 GHz Quad-Core AMD Opteron 8378 (Shanghi) Processor with 6 MB on-chip cache and 64 GB of ECC memory

        Supermicro A+ series motherboard and chassis

        One 160 GB SATA, enterprise quality, Western Digital (WD1600SD) disk drive , and a 5 disk RAID5 configured with 3.1 TB of temporary local disk space accessible as $TMPDIR under PBS and with 64 GB of swap space.

 

One large memory Compute Node consisting of the following:

        eight processor, 1.9 GHz Quad-Core AMD Opteron 8347 HE (Barcelona) Processor with 2 MB on-chip cache and 128 GB of ECC memory

        Supermicro A+ series motherboard and chassis

        One 160 GB SATA, enterprise quality, Western Digital (WD1600SD) disk drive , and a 5 disk RAID0 configured with 4.0 TB of temporary local disk space accessible as $TMPDIR under PBS and with 64 GB of swap space.

 

4 Storage Nodes (shared with Lightning) with each node consisting of the following:

        dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache

        8 GB of ECC memory

        Supermicro A+ series motherboard and chassis with 760 watt, hot swappable, triple-redundant power supplies

        With a total of 60 TeraBytes of usable RAID-6 storage on 87 Western Digital enterprise quality, SATA disk drives .

 

Communication Networks Interconnecting Nodes

        A Mellanox 36-port InfiniBand switch with Mellanox IB cards is used for MPI communication providing low latencies and high bandwidth.

        Four stacking, 48-port managed Gigabit Ethernet switches are used for I/O, management, and PBS communication.

        The cluster has a Gigabit Ethernet connection to the campus network.

 

Security

The cluster can be accessed only by ssh from machines that are on campus and not from off campus. To access lightningsmp from off campus, one must first logon to a machine on campus and then ssh to lightningsmp.