Cersat BigData & Cloud Platform : Hardware

Overview

Following the BigData & Cloud Design Insight for our platform requirements (ie. cheap hardware, horizontal scalability, fault tolerance...), the Cersat platform is quite basic on hardware point of view. There are 3 different hardware components :

100 x servers for both storage and processing
20 x servers for specific services (main schedulers and masters, VmWare ESX, other identified applications ...)
6 x network switches (with a lot of spaghetti links ;-) )

Storage and processing nodes

Horizontal scalability in mind, each server should contain a lot of high capacity disks (cheap) and good processing capability. Over time, models evolved on constructor side, but philosophy stays identical.

We use about 100 rackable 2U servers, each containing :

up to 12 disks 3.5' (2TB to 6TB each) for distributed storage
12 to 16 cores (on 2 cpus)
24GB to 128GB memory (memory needs increased over time, to avoid beeing the bottleneck for distributed processings)

Thanks to our partnership with Dell, we could purchase the following hardware at reasonable prices :

10 x Dell R510   (2*6 cores,  24GB memory, 10x2TB RAID)
27 x Dell R510   (2*6 cores,  48GB memory, 12x3TB RAID)
28 x Dell R720xd (2*6 cores,  48GB memory, 12x4TB RAID)
27 x Dell R720xd (2*6 cores,  64GB memory, 12x4TB JBOD)
 9 x Dell R720xd (2*8 cores, 128GB memory, 12x6TB JBOD)

TODO : Remercier amaury, jean-pierre etc ?

Note about RAID5/6 vs JBOD : one major change over time is that we switched from RAID5 controllers to JBOD controllers. When we started, JBOD controllers were curiously not available on the target models, but is now a more common option (thanks to Cloud Object Storage providers mainly). JBOD is recommended for our MooseFS file system since more efficient and easier to manage than RAID option.

TODO : liens vers mooseFS ? Decrire ce qu'est le JBOD vs RAID ?

Servers for specific needs

29 Dell R410 (1x6 cores, 24GB memory, 3x4TB JBOD)
 2 Dell R620 (1x4 cores, 128GB memory) : for MooseFS master controller

Note : most of the Dell R410 servers were initially used as processing and storage nodes. It was a mistake, but at a certain point we wanted more Gigabit links and nodes... these R410 are now mainly used as service nodes, although some of them are still in the processing & storage grid.

Network

There is 2 distinct networks on the platform :

Main usage network
IDrac / administration network

Main usage network

When starting the project, it was not planned to have more than a few dozen nodes. A requirement was to set up a flat network, each node connected to the others using Gigabit links, with a full speed internal switch bandwith for all nodes.

We choose to use stackable switches, and to easier integration on Ifremer network, we started with 1 Cisco XXXXXXXXXX 48 ports, and added over time 2 others (stacked)

Note : due to stack limiton this model, a maximum of XXX nodes can be connected. This is enough in our case.

TODO : quel modèle exactement ?

IDrac / administration network

Added recently, a dedicated network for administration and maintenance purposes, relying on low cost Dell Network Switches (24 and 48 ports) to connect each server having an iDrac management card.

Note : we mainly use this to collect hardware alerts on our Nagios monitoring console, to have a better hardware failure management.