General Description
lightning is an AMD Opteron cluster with 1056 processor cores on 177 nodes with nodes having either 8 or 64 GBytes of memory (3038 GB total memory on the cluster). Nodes are interconnected with a high performance InfiniPath HTX communication network for MPI communication and a Gigabit Ethernet switch for I/O, management, and PBS communication. 50 Terabytes of RAID-6 storage are integrated into the cluster. The peak performance of each processor core is 4.8 Gflops so 1056 cores gives a peak performance of 5,067 gigaflops (5.067 teraflops). The cluster was purchased from Atipa, see www.atipa.com
Front End Node
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache and 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with two 500 watt, hot swappable, redundant power supplies
Two mirrored, 160 GB SATA, enterprise quality, Western Digital (
·
144 Compute Nodes with each node consisting of the following:
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache and 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with a 500 watt, cold swappable power supply
One 160 GB SATA, enterprise quality, Western Digital (140 GB of temporary local disk space accessible as $TMPDIR under PBS and with 16 GB of swap space.
29 Additional Compute Nodes with each node consisting of the following:
· quad processor, quad core 2.4 GHz AMD 8378 Opteron with 6 MB on-chip cache and 64 GB of ECC memory
· Supermicro A+ series motherboard and chassis with two 500 watt, hot swappable power supplies
A RAID-0 striped across of five enterprise quality Western Digital disk drives configured as 4.5 TB of temporary local disk space accessible as $TMPDIR under PBS and with 128 GB of swap space.
3 Storage Nodes with each node consisting of the following:
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache
· 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with 760 watt, hot swappable, triple-redundant power supplies
· 4 TeraBytes of usable RAID-6 storage with thirteen 400 GB Western Digital (WD4000YR) enterprise quality, SATA disk drives
1 Storage Node with each node consisting of the following:
· dual processor, dual core 2.4 GHz AMD 280 Opteron with 1 MB on-chip cache
· 8 GB of ECC memory
· Supermicro A+ series motherboard and chassis with 760 watt, hot swappable, triple-redundant power supplies
· 38 TeraBytes of usable RAID-6 storage with twenty 750 GB Western Digital (WD5000YR) enterprise quality, SATA disk drives and an expansion box containing 26 TB
Communication Networks
Interconnecting Nodes
· A Mellanox 144-port InfiniBand switch with InfiniPath HTX cards is used for MPI communication providing very low latencies (1.29 microseconds) and very high bandwidth (954 Mbytes/second).
· A Mellanox 36-port InfiniBand switch for MPI communication between the twenty-nine 16 processor core nodes
· Four stacking, 48-port managed Gigabit Ethernet switches are used for I/O, management, and PBS communication.
· The cluster has a Gigabit Ethernet connection to the campus network.
Security
The cluster can be accessed only by ssh from machines that are on campus and not from off campus. To access lightning from off campus, one must first logon to a machine on campus and then ssh to lightning.