Saturday, September 27, 2008

Borg Server Upgrade + Nightmares!

Sometimes, newer is not always better.

I decided I'd upgrade my old home server, Borg, to newer hardware, and with the price of sata being very low, add tons of expandability so I can make some raid arrays. Now Borg is a "all eggs in one basket" type server. It does domain services, file, VMware, DNS, email, some custom applications, and it even powers the pump for the kitchen sink drainage system. (ok, maybe not the last part)

Went to Tigerdirect.ca and looked at prices for motherboards, cpus, ram, and hard drives (only needed 2 as I could reuse two of my satas that are raid 1 in the old server). "Prices are just too good not to upgrade" I figured.

Things really looked good. *add to cart* *search* *add to cart* Oh right, I'll need a bigger power supply too *add to cart* and come to think of it I'll need this, and that... *add to cart* ... oh wow, $1,700, a bit more then I hoped, but this will be worth it. *check out*

Later on the parts arrive...



And the cat was happy too. It's like Christmas!




I also got two of these for the drive arrays with intention of buying a disk controller in the future (which I ended up buying now just to save trouble in the future)





Once all put together I decided to power on and see if it boots, or if there are sparks, or if something is terribly wrong and 25,000 volts arcs from the PSU and kills me.



It POSTED. Now What I expected to see was a disk boot failure of some sort, given there was no OS.

Black screen with blinking cursor...

Reboot!

Mem test freezes half way

Reboot!

Wont power on, keeps turning itself on, then off, then on

Reboot!

Does same but then boots into black screen again

Ok time for memtest.

Boots on it, its ok... not the ram. Hmm well lets try to boot off a CD, maybe that black screen is just equivalent to the boot error.

Ignores the CD (even though memtest worked, and was a CD).

Hmmm ok weird. Anyway I spent a few days, turned out the IDE controller did not work so I went to the store to try and find a sata dvd rom drive. No luck, so I ordered it along with some Y molex cords which I realized I'd need.

A week later I get the molex cords but not the drive. They might of screwed up, I figure I'll call them later and check. I forgot one store, so I go check and they have sata dvd drives so I go pick one up go to install it. Ok weird, only some of the sata ports work. But I do manage to get it to boot and install CentOS 5.2 on the HDD.

So far so good.

Ok, time to start configuring stuff. Set the IP and stuff, do a few other things, reboot. hmm IP wont stick, and... wtf, the nic is dead!

It would pickup the nic, but it would eventually die off 5 minutes into the OS.

Ok, that's it, this MSI motherboard is crap, RMA! Undo everything, repackage it, send it back etc.... So I order an intel board instead, given the cpu is an Intel Core2Quad it should be pretty stable right?

When I sent the mobo off I realized I forgot to remove the plastic X for the heatsink. Oh great, where in the world will I find one of those? I end up contacting Masscool to see if I can get one. They are based in China so I was not sure if they understood my problem or if they were even willing to send one. It was after all, my own mistake. They seemed to be sending one but was still unsure.

A week or so later I get the mobo (AND that sata dvd drive, I have two now!) so decided to just buy another heat sink given I was waiting for a plastic X so I can actually build the server.

Another thing I also noticed the leds on those HDD enclosures are kinda messed up...



Some light up weird like that (notice not the full HDD image is lit) Some don't light up at all. Not consistent. They DO flash though, like when there's hdd activity the orange led gets brighter. But imo it should be OFF when no activity, not half off on some drives and off on others, but anyways....


So I install it and set everything up.

Ok first problem I notice when I boot up is the 3 pin fan connectors and the cpu 4 pin connector does not have enough voltage. [b]I WANT MORE POWER![/b] This is a server. I want 6000 RPM 120 fans, and I want tons of noise!

Bios does not have much options for it. I realize it goes by temp, but geeze, don't wait till my cpu is 100C before giving an extra 2 volts to the fan that's turning slower then my eyes can follow the blades.

I get the solder iron, scissors, and a glue gun. This is serious business.

So I spliced the two case fans and cpu fan to connect to a regular molex connector. For the case fans it went well, for the cpu fan, well lets just say I did not realize it ran on 5 volts, not 12. That smoke looked pretty cool though.

I REALLY don't want to undo the whole mobo again so I can change the fan, which uses the same screws that go through the mobo. (I hate Intel's heat sinks for that - need to remove the mobo to remove the heat sink)

So I break off the blades of the fried fan to clear air flow and glue gun another fan on top, and this one has speed, and runs on 12 volts. Glue gun is FTW. You'd think it would melt inside a PC, but its not that sensible to heat as some may think. I've glue gunned a fan directly on a heat sink before and it never fell off (except for when I forgot to turn it on!) In this case I'm gluing it on the old fan so its glue on plastic. Pretty solid. Most of my mods are ghetto like that. If it works, it's good enough for me!

I go to install CentOS 5.2 but turns out it does not like that hardware for w/e reason so ended up going with Fedora Core 9. As I feared, VMware server 1.0 did not install so had to go with VMware server 2.0 RC2. I'm not a huge fan of server 2.0, I prefer the basic client of 1.0. The next day the release non beta version of 2.0 is released! What luck, so I go ahead and install that. Configured mail and all that stuff, took a few days and everything seems to be running.



That's also with the two sata cards I bought which I had a little fun with, their instructions are not exactly, English. I managed to figure it out and got the 4 ports to work. The 4 on the back don't, its one or the other. At this point I had already done most of the software side of things. That was a whole other nightmare, but configuring linux mail server, samba, dns and stuff all at once is usually always a nightmare. But got it working fairly well. The 1.8TB raid array is quite nice too.

**random cat picture**



Now today was an interesting day. At this point the server config is pretty much done and it's running live. I deployed the server to it's rightful location as it was sitting in my room this whole time, generating noise and heat. It's in the minuses out so the window being opened helped.

I had to unjoin and rejoin the two PCs to the domain. Dealing with the profile issues is a pita, but managed to get my profile to pickup again. Redoing my windows profile would be a pure nightmare. So much stuff to configure.

I go to start coding on one of my projects, and it so happened a backup job kicked in. Normally this is not an issue, but something with this server, its not good at multitasking. The backup job set the load to 4.0 :O Now my program, which is a basic server app that just listens for connections and writes the packets ti disk, skyrocketed the load to 20! I've never seen a server load go even past 2 without me doing something very abnormal like powering on 5 VMs that are DoSing each other.

I decided to reboot to see what happens and it *seems* to have solved it. Load is now 0, and goes to around 5ish when my application has 100's of connections blasting data. To be expected.

I still got some nitty gritty stuff to do like setup the UPS software and ensure all backups are working properly, but in general, it's DONE DONE and DONE!


Ready for deployment:



Deployed:

Yeah its a weird setup. My parents did not go for my server rack idea in the basement. Maybe when I have my own house. :)


Specs:
Fedora Core 9 64-bit edition
Intel Core2Quad 6700 @ 2.66Ghz
8GB of OCZ ram
OS drive: 250GB
Data drive: 3x 1TB Drives in software raid 5 configuration - total of 1.8TB of space

Spare parts:
1 dvdrom drive (was impatient since order did not come in so found one in town)
1 PCI video card (the replacement board has onboard video)
1 plastic X retainer and heat sink (yes Masscool DID send it, was nice of them, did not cost me anything)

price:
probably a little over 2 grand... I lost track. I often had to go to store to get stuff.

I plan to sell the old server as well as my backup server (and the UPS that goes with it) to make up for some of the money. I'm saving up for a house and I spent way too much on this!

Moral of story is, if you don't NEED to upgrade, save your money.

In the end though, it was sorta worth it. Rather then all the /data1 /data2 /data3 drives I have /raid1 and everything is more organized, and I'm running a later OS that is easier to get support for, and the yum repositories work! (FC5 was unsupported so I was unable to use yum to install/update anything)

Next time I build a server I may just buy a dell or other pre-made one. At least that takes away the hardware hassles, and if something breaks they have a pretty good warranty.

No comments: