Friday, July 17, 2009

Raid 0 + medical data + disk failure = EPIC FAIL

I work as a contractor for the hospital, and some server we don't have control over or support failed. I was asked to go take a look at it despite the fact that it's an unknown and unsupported environment (family doctor office not part of actual Hospital infrastructure).

So I check it out, and it's an old dell power edge with internal IDE drives that are hooked to a raid card. This is basically just a normal desktop computer with a fancy case and a raid card. When it boots up it just beeps loudly, and I isolated the beep to be the raid card and not the motherboard, then managed to get it to POST so I checked the raid settings and sure enough a disk failed. So at first I figured no biggie, there must be an option to push the disk out of the raid 1 array and run it as a degraded array until a new drive is added to rebuild. Then I realize, it's not a raid 1, it's a raid 0!!! WTF?!?! Who the hell makes a raid 0 on a server containing medical records?

I go see the doctor about it and he says he has backups, now the IT manager drags me further into this and wants me to try to rebuild his environment from the backups.

Now imagine this: You're a contractor, someone brings you to their house that had a fire and tells you to build it the way it was. That's exactly what this IT manager is asking from me. Backups or not, if the one who setup that environment is not around and can't support it, then tough luck, not much can be done. There is zero documentation or anything on this server and I'm not aware of how it was setup. i can only make speculations based on the rest of the environment such as the fact that DHCP stopped working and DNS is set to that server's IP, or at least, what I assume is that server's IP. Small things like that can give me small clues, but that's it.

It's been about a week now and I'm still trying to get it to work from the backups. The backups appear to be somewhat corrupted or incomplete (may of tried to backup files that were in use such as the DBs) so the restore ain't working well. The IT manager is on my back more and more as time progresses and is extremely agitated that I can't get it to work. What do you expect me to do, pull a 100% working backup out of my ass? Not only that but he expects me to rebuild the environment like it was with all this (unknown to me) programs. Give me a break. Seriously this guy is a nut case and is very degrading. He can GFH.

I took two half days off as I had mold in my house to remove, but part of the reason I took those half days off was to limit the time I'm stuck with him pestering and attacking me.

Is it illegal to release poisonous spiders in his office? My new house has plenty to spare.

No comments: