lightning3: Detailed Description

General Description

lightning3 is an AMD Opteron cluster currently with 672 processor cores (17 compute nodes) with 16 nodes having 256 GBytes of memory and one having 512 GBytes of memory (4608 GB total memory on the cluster). Nodes are interconnected with an Infiniband communication network for low latency/high bandwidth MPI communication and a 1Gigabit Ethernet switch for I/O, management, and PBS communication. Lightning3 has two fileservers with a combined 213 TB of RAID-6 storage which are connected by a 10 Gigabit uplink to the cluster's Gigabit Ethernet switch. The cluster was purchased from Atipa, see http://www.atipa.com

 

Front End Node

        Dual processor, 2.3 GHz Eight-Core AMD Opteron 6134 Magny-Cours Processor with two 6 MB on-chip caches and 32 GB of ECC memory

        Supermicro A+ series motherboard and chassis with two 1000 watt, hot swappable, redundant power supplies

        Two mirrored, 2TB SATA, enterprise quality, Western Digital disk drives

        Users logon, compiler, and launch batch jobs to compute nodes from the Front End Node.

 

17 Compute Nodes with the nodes consisting of the following:

        quad processor, 2.3 GHz Eight-Core AMD Opteron 6134 (Magny-Cours) Processor with two 6 MB on-chip caches and 256 GB of ECC memory (5 nodes) , or

        quad processor, 2.3 GHz Eight-Core AMD Opteron 6134 (Magny-Cours) Processor with two 6 MB on-chip caches and 512 GB of ECC memory (1 node), or

        quad processor, 3.0 GHz Eight-Core AMD Opteron 6220 (InterLagos) Processor with a 16 MB on-chip L3 cache and 256 GB of ECC memory (7 nodes)

        quad processor, 2.3 GHz Sixteen-Core AMD Opteron 6376 (InterLagos) Processor with a 16 MB on-chip L3 cache and 256 GB of ECC memory (4 nodes)

        Supermicro A+ series motherboard and chassis

        One 2 TB SATA, enterprise quality, Western Digital disk drive with 1.6 TB of temporary local disk space accessible as $TMPDIR under PBS and with 4 GB of swap space.

 

3 Fileservers with each fileserver consisting of the following:

        dual processor, quad core 2.4 GHz Intel E5620 Xeons with 12 MB on-chip cache

        24 GB of ECC memory

        Supermicro A+ series motherboard and chassis with 1000 watt, hot swappable, dual-redundant power supplies

        With a total of 213 TeraBytes of usable RAID-6 storage on 120 Seagate enterprise quality, SAS disk drives .

 

Communication Networks Interconnecting Nodes

        A Mellanox 36-port InfiniBand switch with Mellanox IB cards is used for MPI communication providing low latencies and high bandwidth.

        A 48-port managed Gigabit Ethernet switch which connects the nodes and is used for I/O, management, and PBS communication.

        One 8 port 10 Gigabit switch which connects the head node, the two fileservers and a 10 Gigabit uplink for the 48-port managed Gigabit Ethernet switch.

        The cluster has a 10 Gigabit Ethernet connection to the campus network.

 

Security

The cluster can be accessed only by ssh from machines that are on campus and not from off campus. To access lightning3 from off campus, one must first logon to a machine on campus and then ssh to lightning3 or install ISU's VPN software on your machine, so that you can login directly via the Virtual Private Network.