Cluster Theory: Difference between revisions
| (16 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
You require a cluster to run molecular docking at scale. | You require a cluster to run molecular docking at scale. | ||
= Non-Scalable Cluster = | |||
You can just take a bunch of computers, new or used, and an ethernet switch and put them together to make a cluster. That's what we used to do. As the cluster scales, this approach becomes unwieldy and certainly labor intensive to maintain. If you plan to scale and you can afford it, we recommend starting with a scalable design, below. | |||
{{TOCright}} | |||
= Minimalist Scalable Cluster - One computer - $20K = | = Minimalist Scalable Cluster - One computer - $20K = | ||
You can begin a scalable cluster with as few as one computer, albeit a biggish one. | You can begin a scalable cluster with as few as one computer, albeit a biggish one. | ||
We recommend using a hypervisor so you can run multiple virtual machines on a single computer. | We recommend using a hypervisor so you can run multiple virtual machines on a single computer. | ||
You can get away without a rack, just put the machine in a sound-proof room. | You can get away without a rack, just put the machine in a sound-proof room. We would buy: | ||
We would buy: | |||
* a) 1 x 128 core / 256 GB / 16 TB disk @ $20K | * a) 1 x 128 core / 256 GB / 16 TB disk @ $20K | ||
* b) a workstation, but you could even use a laptop | |||
Plug it in to the network and the power supply. That's it. That's all you need to get started. | |||
= Scalable Cluster - $40K = | = Starter Scalable Cluster - $40K = | ||
If you can afford it, starting with a few more computers will be easier to scale up. | If you can afford it, starting with a few more computers will be easier to scale up. This cluster would be an excellent resource and a good foundation for managed growth for a new group getting started. We would buy: | ||
We would buy: | |||
*a) 1 x 96 TB NAS server, $18K | *a) 1 x 96 TB NAS server, $18K | ||
*b) 1 x 128 core / 512 GB @ $20K | *b) 1 x 128 core / 512 GB @ $20K | ||
*c) switch, cables, economy workstation: $2K | *c) switch, cables, economy workstation: $2K | ||
You do not need a rack just yet, but you need a sound-proof room with normal room air conditioning. If you add any more computers to this configuration, you'll need a rack. | |||
* 2 x 15A circuits. | * 2 x 15A circuits. | ||
| Line 27: | Line 30: | ||
*e) spare parts (disks, switch) $4K | *e) spare parts (disks, switch) $4K | ||
*f) ssh portal machine / firewall $1K | *f) ssh portal machine / firewall $1K | ||
Sound proof room, 1 T of cooling = 12,000 Btu/h = 3.516 KW | |||
* 30 A @ 208V (e.g. 1 x L6-30) | * 30 A @ 208V (e.g. 1 x L6-30) | ||
This is a natural progression from the starter cluster above, and can be grown towards the larger cluster below. | |||
= Bigger Scaleable Cluster - $250K = | |||
If you had around $250K to spend, we would buy: | If you had around $250K to spend, we would buy: | ||
*a) 2 x 96 TB NAS server @ $18K = $36K | *a) 2 x 96 TB NAS server @ $18K = $36K | ||
| Line 38: | Line 42: | ||
*e) spare parts (disks, switch) $4K | *e) spare parts (disks, switch) $4K | ||
*f) ssh portal machine / firewall $1K | *f) ssh portal machine / firewall $1K | ||
Sound proof room, 5 T of cooling | |||
* 90 A @ 208V (e.g. 3 L6-30s) | * 90 A @ 208V (e.g. 3 L6-30s) | ||
= Currently... = | |||
We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP. | |||
We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months. | |||
For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000. Running flat out it would pull 28 kW and need 10 T of cooling to be safe. Amazing density at commodity prices. | |||
Back to [[So you want to set up a lab]]. | |||
[[Category:Sysadmin]] | [[Category:Sysadmin]] | ||
[[Category:Theory]] | |||
Latest revision as of 21:09, 18 March 2014
You require a cluster to run molecular docking at scale.
Non-Scalable Cluster
You can just take a bunch of computers, new or used, and an ethernet switch and put them together to make a cluster. That's what we used to do. As the cluster scales, this approach becomes unwieldy and certainly labor intensive to maintain. If you plan to scale and you can afford it, we recommend starting with a scalable design, below.
Minimalist Scalable Cluster - One computer - $20K
You can begin a scalable cluster with as few as one computer, albeit a biggish one. We recommend using a hypervisor so you can run multiple virtual machines on a single computer. You can get away without a rack, just put the machine in a sound-proof room. We would buy:
- a) 1 x 128 core / 256 GB / 16 TB disk @ $20K
- b) a workstation, but you could even use a laptop
Plug it in to the network and the power supply. That's it. That's all you need to get started.
Starter Scalable Cluster - $40K
If you can afford it, starting with a few more computers will be easier to scale up. This cluster would be an excellent resource and a good foundation for managed growth for a new group getting started. We would buy:
- a) 1 x 96 TB NAS server, $18K
- b) 1 x 128 core / 512 GB @ $20K
- c) switch, cables, economy workstation: $2K
You do not need a rack just yet, but you need a sound-proof room with normal room air conditioning. If you add any more computers to this configuration, you'll need a rack.
- 2 x 15A circuits.
Small Scaleable Cluster - $92K
If you had around $100K to spend, we would buy:
- a) 1 x 96 TB NAS server, $18K
- b) 3 x 128 core / 512 GB @ $20K
- c) barebones rack, switch, cables: $1K
- d) 4 workstations @ $2K = $8K
- e) spare parts (disks, switch) $4K
- f) ssh portal machine / firewall $1K
Sound proof room, 1 T of cooling = 12,000 Btu/h = 3.516 KW
- 30 A @ 208V (e.g. 1 x L6-30)
This is a natural progression from the starter cluster above, and can be grown towards the larger cluster below.
Bigger Scaleable Cluster - $250K
If you had around $250K to spend, we would buy:
- a) 2 x 96 TB NAS server @ $18K = $36K
- b) 10 x 128 core / 512 GB @ $20K = $200K
- c) barebones rack, switch, cables: $1K
- d) 4 workstations @ $2K = $8K
- e) spare parts (disks, switch) $4K
- f) ssh portal machine / firewall $1K
Sound proof room, 5 T of cooling
- 90 A @ 208V (e.g. 3 L6-30s)
Currently...
We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months.
For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000. Running flat out it would pull 28 kW and need 10 T of cooling to be safe. Amazing density at commodity prices.
Back to So you want to set up a lab.