System Capabilities

Network

Nodes with the QDR InfiniBand (IB) option will plug into 32-port IB switches, allowing up to 512 cores to communicate at full bisection bandwidth for low latency, parallel computing. 

Storage

TSCC users will receive 100GB of home file storage and shared access to the >200TB Data Oasis Lustre high performance file system.  (There is a 90-day purge policy on Data Oasis for normal usage.)  Additional persistent storage can be mounted from lab file servers over the campus network or can be purchased from SDSC.

Usage Model

Each year, condo cluster participants receive an amount of cluster runtime proportional to the capabilities of their purchased nodes. For example, a participant that purchases eight general computing nodes will receive just under 1.1 M Service Units (1SU=1core-­‐hour) of time, which represents 24x365 usage of 128 cores, allowing for 3% maintenance downtime. These core-­‐hours can be used any time during the year on any of the computing nodes; however unused core-­‐hours by condo participants expire at the end of each year.

Condo participants' jobs that require a number of cores less than or equal to their purchased nodes are guaranteed to start within eight hours of submission and can run for an unlimited amount of time. Jobs that extend into the cluster hotel nodes have a 72-­‐hour time limit and share a queue with the jobs of pay-­‐as-­‐you-­‐go users, while jobs that extend to other participants’ condo nodes have an eight-­‐hour time limit. Condo participants may submit gleaning jobs to run on idle computing nodes. These jobs are not charged against the submitter's SU balance, but they may be terminated at any time by the scheduler if the nodes where they are running are needed to run higher-­‐priority jobs.

Because the capabilities and purpose of the GPU nodes differ significantly from the general computing nodes, SUs received for contributed GPU nodes and general computing nodes cannot be interchanged.

Pay-­‐as-­‐you-­‐go (hotel) users’ jobs can only run on the hotel nodes. Initially there will be 40 general computing nodes (6,400cores); additional nodes may be added based on demand. Hotel nodes are configured with 64 GB of memory and an IB interface. The general computing nodes will be allocated per-­‐core, allowing up to 16 jobs to run on each node simultaneously. Hotel jobs can also run on the large-­‐memory PDAF nodes; the PDAF nodes will allow a maximum of two jobs per node (i.e.,256GB or 512GB of memory), meaning that use of either 16 or 32 cores will be charged to each job.

Plan

Participants who contribute to the cluster will have priority access to the nodes that they contribute. In addition, they have the option to run jobs on additional cluster nodes when available, effectively increasing their computing capability and flexibility. The TSCC allows participating researchers access to additional cycles during times of peak use, through a model that pools computing resources. During times of intense research, this approach provides participants with far more computational power than they would have if running only on their own hardware. It also supports running jobs at a higher core count than when restricted to their own nodes.

The condo purchaser has the option of taking possession of their nodes at any time; however once equipment is removed, it cannot later be returned to TSCC.

After the expiration of the three-year warranty, condo participants may leave their contributed nodes in the TSCC for another year, as long as the nodes remain operational.  At the end of four years, participants must take possession of or surplus their equipment.