Dec 6: Getting the most out of hardware

This is the sixth post in the FastMail 2015 Advent Calendar. Stay tuned for another post tomorrow.

This post is about technical stuff, because that’s what I do, but it’s relatively
short and simple compared to my usual posts!

Being able to provide the quality of service that we do for such a low price is partially about big things like major efficiency improvements to our server software, but it’s also about optimising small things.

Standalone IMAP servers

I blogged about our servers in last year’s advent calendar. If you look at the section for one of the older servers (imap14) you’ll notice that the search partition was less than 50% full, yet the spool
partitions were quite full. This pattern was repeated across all our 2U servers.

The OS partitions were also 1TB and had tons of free space. This was carved out of the 16TB RAID6, leaving room for only 15 slots.

In the process of rebuilding machines for Amsterdam, we decided to make use of that
free space, and reconfigure them, so instead of the RAID being:

RAID r6:[ 0 1 2 3 4 5 6 7 8 9 ] r1:[ 10 11 ]

VOLUME [ 1TB, 15TB ] [ 2TB ]

The RAID1 has been moved to the first two disks, and OS is shared with search instead of spool:

RAID r1:[ 0 1 ] r6:[ 2 3 4 5 6 7 8 9 10 11 ]

VOLUME [ 500Gb 1.5TB ] [ 16TB ]

Giving 16 slots instead of 15. It also means that the OS is shared with search rather than spool. With our patterns of IO usage, this makes the OS more responsive. So a machine now looks like:

/dev/sda1                  15995848   2393272  12783376  16% /
/dev/sda5                 462344068   9734232 429101052   3% /local
/dev/mapper/sdb1         1441724500 736678888 705029228  52% /mnt/si10search
/dev/mapper/sdc1          961104216 748308608 212779224  78% /mnt/si10t01
/dev/mapper/sdc2          961105240 793163232 167925624  83% /mnt/si10t02
/dev/mapper/sdc3          961105240 812993724 148095132  85% /mnt/si10t03
/dev/mapper/sdc4          961105240 835672212 125416644  87% /mnt/si10t04
/dev/mapper/sdc5          961105240 820427704 140661152  86% /mnt/si10t05
/dev/mapper/sdc6          961105240 789696468 171392388  83% /mnt/si10t06
/dev/mapper/sdc7          961105240 821484964 139603892  86% /mnt/si10t07
/dev/mapper/sdc8          961105240 737563548 223525308  77% /mnt/si10t08
/dev/mapper/sdc9          961105240 759814248 201274608  80% /mnt/si10t09
/dev/mapper/sdc10         961105240 821480792 139608064  86% /mnt/si10t10
/dev/mapper/sdc11         961104216 787875416 173212416  82% /mnt/si10t11
/dev/mapper/sdc12         961105240 754964180 206124676  79% /mnt/si10t12
/dev/mapper/sdc13         961105240 765869784 195219072  80% /mnt/si10t13
/dev/mapper/sdc14         961104216     73364 961014468   1% /mnt/si10t14
/dev/mapper/sdc15         961105240     73364 961015492   1% /mnt/si10t15
/dev/mapper/sdc16         961103192     73364 961013444   1% /mnt/si10t16
/dev/md0                  384320508 106689472 277614652  28% /mnt/si10ssd

(search and spool partitions are encrypted with LUKS, the SSDs have on-drive encryption, so they’re just software RAID1 without OS level encryption as well)

We’re slowly moving all the old machines over to this layout. Right now imap24 and imap29 have been reconfigured, and are syncing up new replicas from imap25 and imap26. That should be finished in about a week, then I’ll empty those two servers and reconfigure the drives.

There are almost always long term migrations like this happening in the background. The latest IMAP hardware has 4TB big drives and 800GB SSDs along with 192GB RAM and some serious CPU, and we add new servers every few months and move users to them - but it’s still good to get the best we can out of the older machines.