Acquire and deploy hardware - Revision history

Frodo: /* Setting up a network */

2014-09-22T18:29:04Z

Setting up a network

← Older revision		Revision as of 18:29, 22 September 2014
Line 7:		Line 7:

	= Setting up a network =		= Setting up a network =
	Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch~~, but you shouldn't~~ have to, ~~unless~~ you ~~are in~~ a ~~very funky networking environment.~~		Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch (compatibility with and isolation from a complex networking environment). You should not have to buy the fancy $2500 switch. Suggest starting with the $250 switch, and then upgrade if you must (and now you've got a backup switch!)

	Network topology will depend on your circumstances. We recommend running a private network, and keeping any machines that don't have to be on the public network off it for security. We recommend using a portal computer and ssh keys for remote access.		Network topology will depend on your circumstances. We recommend running a private network, and keeping any machines that don't have to be on the public network off it for security. We recommend using a portal computer and ssh keys for remote access.

Frodo: /* Physical setup */

2014-03-18T16:22:45Z

Physical setup

← Older revision		Revision as of 16:22, 18 March 2014
Line 18:		Line 18:

	=Physical setup =		=Physical setup =
	You can fit everything in one rack. Put the switch in the middle.		You can fit everything described here in one rack. You only need a rack when you buy the third computer. Put the switch in the middle for neater cabling.

Frodo at 16:18, 18 March 2014

2014-03-18T16:18:11Z

← Older revision		Revision as of 16:18, 18 March 2014
Line 1:		Line 1:
	Here is our recommendation of how to build a computer cluster that will work well for molecular docking and cheminformatics. This document can be used whether you already own hardware, or whether you are planning to buy new. ~~All recommendations are the best we know as of Feb 2014. Things do change, but this advice should be ok through 2015.~~ This is part of a series of articles called [[So you want to set up a lab]].		Here is our recommendation of how to build a computer cluster that will work well for molecular docking and cheminformatics. This document can be used whether you already own hardware, or whether you are planning to buy new. This is part of a series of articles called [[So you want to set up a lab]].

	{{TOCright}}		{{TOCright}}

Frodo: /* Buying CPU and disk */

2014-03-18T16:17:41Z

Buying CPU and disk

← Older revision		Revision as of 16:17, 18 March 2014
Line 4:		Line 4:

	= Buying CPU and disk =		= Buying CPU and disk =
	We ~~are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.~~		We describe several sizes of cluster in [[Cluster Theory]].
	~~We recommend buying two different kinds~~ of ~~machines~~ in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months.

	For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000, pull 28 kW and need 5 T of cooling. Amazing density at commodity prices.

	If you are on a budget, you can get a single C6145 or equivalent, less memory, and use the disk in the CPU machine, don't get a separate disk server. There are ways to get a serious computational lab started for as little as $10,000 of hardware, you'll just have to cut a few corners.

	= Setting up a network =		= Setting up a network =

Frodo at 10:05, 13 March 2014

2014-03-13T10:05:28Z

← Older revision		Revision as of 10:05, 13 March 2014
Line 21:		Line 21:
	= If you have more money =		= If you have more money =
	Our recommended CPU machine costs $20K (128 cores). Our recommended disk machine costs $20K (96 TB raw). We recommend just buying these two machines in a proportion that makes sense for your research. A single switch will handle an entire rack. If you have a lot more money, ask us. Don't forget about cooling, UPS and power.		Our recommended CPU machine costs $20K (128 cores). Our recommended disk machine costs $20K (96 TB raw). We recommend just buying these two machines in a proportion that makes sense for your research. A single switch will handle an entire rack. If you have a lot more money, ask us. Don't forget about cooling, UPS and power.

			=Physical setup =
			You can fit everything in one rack. Put the switch in the middle.


	Return to [[So you want to set up a lab]].		Return to [[So you want to set up a lab]].

	[[Category:Sysadmin]]		[[Category:Sysadmin]]

Frodo: /* Setting up a network */

2014-03-13T09:28:53Z

Setting up a network

← Older revision		Revision as of 09:28, 13 March 2014
Line 13:		Line 13:
	= Setting up a network =		= Setting up a network =
	Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch, but you shouldn't have to, unless you are in a very funky networking environment.		Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch, but you shouldn't have to, unless you are in a very funky networking environment.

			Network topology will depend on your circumstances. We recommend running a private network, and keeping any machines that don't have to be on the public network off it for security. We recommend using a portal computer and ssh keys for remote access.

	= Minimum requirements for legacy equipment =		= Minimum requirements for legacy equipment =

Frodo at 09:26, 13 March 2014

2014-03-13T09:26:39Z

← Older revision		Revision as of 09:26, 13 March 2014
Line 3:		Line 3:
	{{TOCright}}		{{TOCright}}

	== Buying CPU and disk ==		= Buying CPU and disk =
	We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.		We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.
	We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months.		We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months.
Line 11:		Line 11:
	If you are on a budget, you can get a single C6145 or equivalent, less memory, and use the disk in the CPU machine, don't get a separate disk server. There are ways to get a serious computational lab started for as little as $10,000 of hardware, you'll just have to cut a few corners.		If you are on a budget, you can get a single C6145 or equivalent, less memory, and use the disk in the CPU machine, don't get a separate disk server. There are ways to get a serious computational lab started for as little as $10,000 of hardware, you'll just have to cut a few corners.

	== Setting up a network ==		= Setting up a network =
	Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch, but you shouldn't have to, unless you are in a very funky networking environment.		Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch, but you shouldn't have to, unless you are in a very funky networking environment.

Frodo: /* Setting up a network */

2014-03-13T09:26:01Z

Setting up a network

← Older revision		Revision as of 09:26, 13 March 2014
Line 12:		Line 12:

	== Setting up a network ==		== Setting up a network ==
	~~$250 for a~~ Managed 24 port GigE switch with VPN support. About $250. ~~Will~~ do for most circumstances. Allows you to run both private and public networks.		Managed 24 port GigE switch with VPN support. About $250. This is not a full layer 3 switch, but it should do for most circumstances. Allows you to run both private and public networks. We do know why people spend $2500 for a full layer 3 switch, but you shouldn't have to, unless you are in a very funky networking environment.

	= Minimum requirements for legacy equipment =		= Minimum requirements for legacy equipment =

Frodo: /* Buying CPU and disk */

2014-03-13T09:24:09Z

Buying CPU and disk

← Older revision		Revision as of 09:24, 13 March 2014
Line 5:		Line 5:
	== Buying CPU and disk ==		== Buying CPU and disk ==
	We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.		We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.
	We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. An amazing development in the last 12 months.		We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. And it is SAS instead of SATA so we only need RAID6 and not RAID10 formatting. An amazing development in the last 12 months.

	For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000, pull 28 kW and need 5 T of cooling. Amazing density at commodity prices.		For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000, pull 28 kW and need 5 T of cooling. Amazing density at commodity prices.

			If you are on a budget, you can get a single C6145 or equivalent, less memory, and use the disk in the CPU machine, don't get a separate disk server. There are ways to get a serious computational lab started for as little as $10,000 of hardware, you'll just have to cut a few corners.

	== Setting up a network ==		== Setting up a network ==

Frodo at 09:20, 13 March 2014

2014-03-13T09:20:50Z

← Older revision		Revision as of 09:20, 13 March 2014
Line 3:		Line 3:
	{{TOCright}}		{{TOCright}}

	== ~~About buying~~ CPU and disk ==		== Buying CPU and disk ==
	We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.		We are currently buying CPU from Silicon Mechanics and Dell and disk from Silicon Mechanics and HP.
	We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. An amazing development in the last 12 months.		We recommend buying two different kinds of machines in a modular fashion: head nodes to which disk enclosures may be attached and cpu nodes which contain large numbers of cores. We are currently buying enclosures holding 12 SAS disks of 4TB each for 48 TB raw for around $8000 or about 6 raw GB per dollar. Formatted RAID6 this works out to 36TB or 4.5 formatted GB per dollar. We like the HP P822 high performance RAID controller. Compare this to what we were paying just a year ago in spring 2013: 25 TB for $10,000 or 2.5 TB per dollar unformatted. An amazing development in the last 12 months.
Line 9:		Line 9:
	For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000, pull 28 kW and need 5 T of cooling. Amazing density at commodity prices.		For CPU, we like the C6145 from Dell. For around $20,000 you get 2 machines in a 2U form each with 64 cores and 256 GB memory and a pair of RAID1 formatted disks each. A single 42U rack could hold 2560 cores and still have room for a switch. Of course, this would cost you $400,000, pull 28 kW and need 5 T of cooling. Amazing density at commodity prices.

	== ~~About setting~~ up a network ==		== Setting up a network ==
	$250 for a Managed 24 port GigE switch with VPN support. About $250. Will do for most circumstances. Allows you to run both private and public networks.		$250 for a Managed 24 port GigE switch with VPN support. About $250. Will do for most circumstances. Allows you to run both private and public networks.