Monday, September 29, 2008

Not so bad after all

Server load stats can be deceiving. I'm running 7 VMs and the system is barely taking any hit.

(click to enlarge + view more)

I played with the drives yesterday and the one that was reporting slow is no longer reporting slow. But now another one is.... I was starting to fear a back plane issue but it turned out to be a drive name mix up so it would not pickup any new drives in the slot I was trying.

Reboot wins again!

Think I can safely leave the server at that. I can deal with this i/o issue considering its only affecting server stats, not so much performance. The backup jobs kill the server though but those happen at night anyway...

So think all is good, and God was with me as I struggled to get this working. It could of been much worse.

Now I need continue to code my hosting control panel. I still need to find a good name for it. It's hard to code something that does not have a name yet. I've been refering to it as "controlpanel" in my code so far. I officially started on it yesterday before I did that hdd swap thing to see if the back plane was at fault.

Sunday, September 28, 2008

Yeah, so looks like I got a failing hard drive...

Problem is, it's not really technically failed. It's just VERY VERY slow. My raid has been performing horrendously so was wondering why. With help of people at Anandtech we ruled that it was /dev/sdc that was slow. That turns out to be one of my newest seagate drives too! SMART is not showing any signs of failures.

So figured I'd rule out the backplane slot so I put it in another slot and proceeded to rebuild the raid. Lets just say that did not go too well... Though I'm fearing it may even be the backplane, I was moving drives around to see and does not look too good.

But either way this is going to cost even more money. This is madness!

Never again am I building my own server. A dell poweredge sounds more affordable, considering you take it out of the box and don't have to spend more money because half the stuff does not work.

Saturday, September 27, 2008

Borg Server Upgrade + Nightmares!

Sometimes, newer is not always better.

I decided I'd upgrade my old home server, Borg, to newer hardware, and with the price of sata being very low, add tons of expandability so I can make some raid arrays. Now Borg is a "all eggs in one basket" type server. It does domain services, file, VMware, DNS, email, some custom applications, and it even powers the pump for the kitchen sink drainage system. (ok, maybe not the last part)

Went to and looked at prices for motherboards, cpus, ram, and hard drives (only needed 2 as I could reuse two of my satas that are raid 1 in the old server). "Prices are just too good not to upgrade" I figured.

Things really looked good. *add to cart* *search* *add to cart* Oh right, I'll need a bigger power supply too *add to cart* and come to think of it I'll need this, and that... *add to cart* ... oh wow, $1,700, a bit more then I hoped, but this will be worth it. *check out*

Later on the parts arrive...

And the cat was happy too. It's like Christmas!

I also got two of these for the drive arrays with intention of buying a disk controller in the future (which I ended up buying now just to save trouble in the future)

Once all put together I decided to power on and see if it boots, or if there are sparks, or if something is terribly wrong and 25,000 volts arcs from the PSU and kills me.

It POSTED. Now What I expected to see was a disk boot failure of some sort, given there was no OS.

Black screen with blinking cursor...


Mem test freezes half way


Wont power on, keeps turning itself on, then off, then on


Does same but then boots into black screen again

Ok time for memtest.

Boots on it, its ok... not the ram. Hmm well lets try to boot off a CD, maybe that black screen is just equivalent to the boot error.

Ignores the CD (even though memtest worked, and was a CD).

Hmmm ok weird. Anyway I spent a few days, turned out the IDE controller did not work so I went to the store to try and find a sata dvd rom drive. No luck, so I ordered it along with some Y molex cords which I realized I'd need.

A week later I get the molex cords but not the drive. They might of screwed up, I figure I'll call them later and check. I forgot one store, so I go check and they have sata dvd drives so I go pick one up go to install it. Ok weird, only some of the sata ports work. But I do manage to get it to boot and install CentOS 5.2 on the HDD.

So far so good.

Ok, time to start configuring stuff. Set the IP and stuff, do a few other things, reboot. hmm IP wont stick, and... wtf, the nic is dead!

It would pickup the nic, but it would eventually die off 5 minutes into the OS.

Ok, that's it, this MSI motherboard is crap, RMA! Undo everything, repackage it, send it back etc.... So I order an intel board instead, given the cpu is an Intel Core2Quad it should be pretty stable right?

When I sent the mobo off I realized I forgot to remove the plastic X for the heatsink. Oh great, where in the world will I find one of those? I end up contacting Masscool to see if I can get one. They are based in China so I was not sure if they understood my problem or if they were even willing to send one. It was after all, my own mistake. They seemed to be sending one but was still unsure.

A week or so later I get the mobo (AND that sata dvd drive, I have two now!) so decided to just buy another heat sink given I was waiting for a plastic X so I can actually build the server.

Another thing I also noticed the leds on those HDD enclosures are kinda messed up...

Some light up weird like that (notice not the full HDD image is lit) Some don't light up at all. Not consistent. They DO flash though, like when there's hdd activity the orange led gets brighter. But imo it should be OFF when no activity, not half off on some drives and off on others, but anyways....

So I install it and set everything up.

Ok first problem I notice when I boot up is the 3 pin fan connectors and the cpu 4 pin connector does not have enough voltage. [b]I WANT MORE POWER![/b] This is a server. I want 6000 RPM 120 fans, and I want tons of noise!

Bios does not have much options for it. I realize it goes by temp, but geeze, don't wait till my cpu is 100C before giving an extra 2 volts to the fan that's turning slower then my eyes can follow the blades.

I get the solder iron, scissors, and a glue gun. This is serious business.

So I spliced the two case fans and cpu fan to connect to a regular molex connector. For the case fans it went well, for the cpu fan, well lets just say I did not realize it ran on 5 volts, not 12. That smoke looked pretty cool though.

I REALLY don't want to undo the whole mobo again so I can change the fan, which uses the same screws that go through the mobo. (I hate Intel's heat sinks for that - need to remove the mobo to remove the heat sink)

So I break off the blades of the fried fan to clear air flow and glue gun another fan on top, and this one has speed, and runs on 12 volts. Glue gun is FTW. You'd think it would melt inside a PC, but its not that sensible to heat as some may think. I've glue gunned a fan directly on a heat sink before and it never fell off (except for when I forgot to turn it on!) In this case I'm gluing it on the old fan so its glue on plastic. Pretty solid. Most of my mods are ghetto like that. If it works, it's good enough for me!

I go to install CentOS 5.2 but turns out it does not like that hardware for w/e reason so ended up going with Fedora Core 9. As I feared, VMware server 1.0 did not install so had to go with VMware server 2.0 RC2. I'm not a huge fan of server 2.0, I prefer the basic client of 1.0. The next day the release non beta version of 2.0 is released! What luck, so I go ahead and install that. Configured mail and all that stuff, took a few days and everything seems to be running.

That's also with the two sata cards I bought which I had a little fun with, their instructions are not exactly, English. I managed to figure it out and got the 4 ports to work. The 4 on the back don't, its one or the other. At this point I had already done most of the software side of things. That was a whole other nightmare, but configuring linux mail server, samba, dns and stuff all at once is usually always a nightmare. But got it working fairly well. The 1.8TB raid array is quite nice too.

**random cat picture**

Now today was an interesting day. At this point the server config is pretty much done and it's running live. I deployed the server to it's rightful location as it was sitting in my room this whole time, generating noise and heat. It's in the minuses out so the window being opened helped.

I had to unjoin and rejoin the two PCs to the domain. Dealing with the profile issues is a pita, but managed to get my profile to pickup again. Redoing my windows profile would be a pure nightmare. So much stuff to configure.

I go to start coding on one of my projects, and it so happened a backup job kicked in. Normally this is not an issue, but something with this server, its not good at multitasking. The backup job set the load to 4.0 :O Now my program, which is a basic server app that just listens for connections and writes the packets ti disk, skyrocketed the load to 20! I've never seen a server load go even past 2 without me doing something very abnormal like powering on 5 VMs that are DoSing each other.

I decided to reboot to see what happens and it *seems* to have solved it. Load is now 0, and goes to around 5ish when my application has 100's of connections blasting data. To be expected.

I still got some nitty gritty stuff to do like setup the UPS software and ensure all backups are working properly, but in general, it's DONE DONE and DONE!

Ready for deployment:


Yeah its a weird setup. My parents did not go for my server rack idea in the basement. Maybe when I have my own house. :)

Fedora Core 9 64-bit edition
Intel Core2Quad 6700 @ 2.66Ghz
8GB of OCZ ram
OS drive: 250GB
Data drive: 3x 1TB Drives in software raid 5 configuration - total of 1.8TB of space

Spare parts:
1 dvdrom drive (was impatient since order did not come in so found one in town)
1 PCI video card (the replacement board has onboard video)
1 plastic X retainer and heat sink (yes Masscool DID send it, was nice of them, did not cost me anything)

probably a little over 2 grand... I lost track. I often had to go to store to get stuff.

I plan to sell the old server as well as my backup server (and the UPS that goes with it) to make up for some of the money. I'm saving up for a house and I spent way too much on this!

Moral of story is, if you don't NEED to upgrade, save your money.

In the end though, it was sorta worth it. Rather then all the /data1 /data2 /data3 drives I have /raid1 and everything is more organized, and I'm running a later OS that is easier to get support for, and the yum repositories work! (FC5 was unsupported so I was unable to use yum to install/update anything)

Next time I build a server I may just buy a dell or other pre-made one. At least that takes away the hardware hassles, and if something breaks they have a pretty good warranty.

Ahhh cobras!

Yeah I decided to make a blog so I can bore people with server and programming stuff.

I'll also post stuff here about my various projects such as and I also have some plans to start a hosting company on the side, but thats still in planning/coding stages.