Cluster Sitsfleisch

Tasty?

My adventure in selecting components and building the 2 node cluster I talked about earlier, needs some fleshing out.  In early 2012, I read How much linux cluster for 6k and was somewhat inspired -- at least by the economy of it.  For aesthetic inspiration I found Helmer (Swedish for cluster?) or the details of justacluster and its use as a render farm for Sintel just really interesting.

Some choices which I made, that may be a matter of taste to some:

• Intel 1Gbe NIC interfaces on the compute node motherboards since they seemed better than the RealTek equivalents (anecdotally) for features and consistency
• Nodes should be thin:  this means they do not rely on local bootable hard disks.  The main advantage of this is avoiding "state" problems (inconsistencies in boot state).  Its bad enough that each node has its own BIOS!
• NFS is good enough to serve software binaries to the compute nodes
• PXE booting for compute nodes
• good central control (issuing commands to all machines)
• Want low latency networking and very low clock slew (PTP is a requirement) so that events happening in the cluster can be properly ordered and measured for our application.  This is probably a key reason why cloud solutions are not so great --  internal jitter hurts a trading platform.

Please bear in mind that prices I quote are from early 2012 and are in Canadian dollars.

For racking I had in mind (as more compute nodes were purchased) a utility shelving rack of the kind you buy on sale and use for light stuff in your garage.  Pricing for something that could hold 15 uATX motherboards would probably run 25$ to 45$ (depending on luck with sales).

Switch choices

Commodity Gigabit switches:

Switch		Spec			Cost($)
Manuf	Model	Capacity	Ports	Extras
D-Link	DGS1210-48	96Gbps	48	4 SFP, DHCP	552.00
TRENDnet	TEG-448WS	96Gbps	48	4 SFP expansion	490.00
Netgear	GS748TNA	40Gbps	48	4 SFP, DHCP	464.00
Cisco	SR2024CT-NA		24	2 SFP	323.91
D-Link	DGS-1100-24		24		247.38
HP	1410-24G	48Gbps	24	2 SFP	318.06
Netgear	GS724T	48Gbps	24	2 SFP	145.00

Trade-off between immediate cost and expandability.

Cables:

• 7' CAT6 patch cable costs 2.10$.
• 2' Universal power cable 3.51$.
• 6 outlet power surge protector 4.97$

Node Choices

Compute power, number of cores, and Watts:

Chip		Specs						Cost ($)
Manuf	Model	Cores	Ghz	GkBnch	L2/3	TDP	maxGB	Chip	MB	Mem/G	Cool	Pwr
Intel	i7-3930K	6	3.2	21.5	12	130	64	655.25	397.76	9.50	103.74	100
Intel	i7-2600K	4	3.4	12.1	6	95	32	359.34	102.35	4.87	30.66	40
Intel	i5-2500K	4	3.3	9.3	6	95	32	239.88	102.35	4.87	30.66	40
AMD	FX-8120	8	3.1	8.77	8	125	32	233.67	107.00	4.87	30.66	50
AMD	X6-1100T	6	3.3	8.26	6	125	32	194.99	107.00	4.87	30.66	50
AMD	X6-1055T	6	2.8	7.39	6	95	32	149.99	107.00	4.87	30.66	40
Intel	Atom D425	2	1.8	1.29	0.5	10	4	89.99	incl	6.50	incl	25

All motherboards (MB) are GBLAN, and accept DDR3 RAM (1333-SODIMM).
GkBnch is the Geekbench benchmark score divided by 1000.

I retrieved this data from various online sources and did not measure it myself, but it seemed to all fit with my pre-conceptions: AMD had less IPC and more power but cost less than Intel (at the time).

Node Summary

How many (fractional) nodes would you get for 6000$ (6K)? How much of everything is there?

Node type	Cores	MMem	CMem	GkB	Cost	Load Pwr	6KNodes	6KCores	6KMMem	6KGkB	6KPwr
i7-3930K	6	32	12	21.5	1567.19	0.228kW	3.8	22.8	121.6	81.7	0.87kW
i7-2600K	4	16	6	12.1	616.71	0.164kW	9.7	38.8	155.2	117.4	1.59kW
i5-2500K	4	16	6	9.3	497.25	0.152kW	10.6	42.4	169.6	98.6	1.61kW
FX-8120	8	16	8	8.77	505.69	0.223kW	12.1	96.8	193.6	106.1	2.70kW
X6-1100T	6	16	6	8.26	467.01	0.229kW	12.8	76.8	204.8	105.7	2.93kW
X6-1055T	6	16	6	7.39	412.01	0.222kW	14.6	87.6	233.6	107.9	3.24kW
Atom D425	2	4	0.5	1.29	147.43	0.036kW	40.7	81.4	162.8	52.5	1.47kW

At 14 cents per kWh, the electrical running cost for 1.0kW cluster would be 1226$ for a year.  Not bad.
Several 15 Amp circuits would be needed for anything with more than 6 nodes, and HVAC needs to be considered where ever it sits.

Comparison to AWS

Ok, the comparison may not be fair in some ways since EC2 is not designed to be a cheap commodity low-latency rig (one wonders what the clock slew is like?).  But people who care about acronyms like EBDA and TCO want to know where the break even is on full duty cycle:

Using the numbers I found at AWSTests, I compared a job we do at my work with: 400 m1.large instances which provide 7.5G RAM, 4 ECU, 3k Geekbench score.  Each of these on-demand instances cost 0.34$ per hour at the time.

From the cluster a single i7-2600k compute node provides (relative to the m1.large instance): 4 times the CPU (ECU or Geekbench) and twice the RAM at 0.04$ for each hour of power.  The cost outlay for the node is 685$.  With these assumptions, running at full tilt the cluster reaches cost parity with AWS after 98 days (but remember - you still own a cluster after running it for 98 days).

Summary

So with a 600$ master computer (commodity PC serving NFS PXE bootable partitions), the whole shebang (with 2 thin compute nodes) costs less than 3000$ Canadian assuming you go with the i7-2600k CPUs for the nodes.  This is what happened for us.  I will try to give more (tasty) detail on the software we run on the master and nodes in  a future post.  Certainly some poison (trial with error) was encountered in that process.  Stay tuned!