Friday, May 17, 2013


For the longest time I've been wanting to build a NAS or SAN (not decided yet how I will set it up) I'll just call it a file server for now.

Basically, I want central storage so any server I have is now disk less other than the OS, and data is centrally stored and backed up. Easier to maintain, and also opens up my options when building or buying a server.

I want to setup a better VM environment, and now that I'll have central storage I don't have to worry about storage at the server level.  Servers with multiple drive bays tend to be more expensive especially when I look at stuff like Dell or even Supermicro.  The other nice thing is by having the raid handled by the storage server I can use a hypervisor without worrying about what kind of software raid it supports.  After working a lot with Linux software raid, I swear by it now.  It is rock solid, and no need to worry about the hardware failure point of raid cards, or having to take a server offline to modify the array. Can do everything live with Linux raid, such as adding more capacity or even changing raid levels (with certain limitations mind you, and kernel has to be new enough).

Without further delay, this is the case:

It is a 24 bay Supermicro CSE-846A-R1200B.

Rest of components as follows:

3x Sata card:
IBM ServeRaid M1015 (will flash to LSI9211-IT)

Intel Xeon E3 1230 V2 Quad Core Processor 3.3GHZ 8MB LGA1155 69W Retail Box

Supermicro MBD-X9SCM LGA1155 C204 DDR3 ECC 6SATA 4PCIE 2GBE IPMI 9USB2.0 mATX Motherboard

Supermicro MEM-DR380L-HL01-EU13 8GB DDR3-1333 240PT 1.5V DIMM CL9 ECC Server Memory

SSD for OS:
Samsung 840 Series 120GB 2.5in SATA3 MDX Solid State Disk Flash Drive SSD

Still waiting for the motherboard and ram... what a tease!  They've been backordered and with my luck it will probably take bloody ages. 

Also bought 4x 3TB hard drives to give me a starting array.  Probably going to do raid 10.  May as well, got plenty of bays.   I will also be moving the 9 1TB drives from my current server to that box.  Since I'm using Linux raid, it will be plug and play and the raid 5 will come up on the new box. 

This whole project is not cheap.  Came in at roughly $3,500!  I originally wanted to cheap out but even cheaping out would have ended up being close to 2k, so figured may as well just step it up and go with a better quality build.  The nice thing with this vs a commercial SAN is the expandability.  I can go buy whatever drives I want and throw them in.  5 years down the line, it will still be useful.  With commercial proprietary stuff once the 3 or 5 year warranty is out, you're pretty much out of luck for parts such as drives.  At one place I worked we had an IBM SAN and they wanted 26 thousand dollars for a part, and it was proprietary so no other alternative.

Once this is paid off I will look at getting perhaps a used server that has lot of ram and a modern cpu with VT technology, so I can setup a separate VM server.  I've been wanting to start playing around with various stuff like server 2008/2010 (Metro on a server, eww)  so I can update my skills. That's my excuse for spending this much cash. :P  Also my current home production setup is really on it's last legs.  It's using a bunch of cheap PCI sata cards to get 10 drives going.  I get lot of drive failures and tons of I/O errors.  Pretty sure it's the cheap sata cards that are the issue.

I have several ideas in mind on how to set this up: 

1: Set it up on an isolated physical network, and have servers have a dedicated NIC to access it. This would more or less make it a SAN, and any future storage I add would simply be tacked on to this network.

2: Set it up as simply a server, making it more a NAS.

#1 sounds fun, but #2 might make more sense for now.  I will most likely team the two nics directly to my main gigabit switch which is managed.  Maybe in the future I can get another managed switch for the SAN.  Either way I will most likely use NFS. iSCSI is a lot of fun too, but with a server this powerful NFS makes sense so I can make the OS take care of the entire FS such as journaling.  Now if that motherboard and ram can come in sooner!  I have a whole long weekend to wait, at very least. 

No comments: