Friday, December 2, 2011

Finally Started my new job!

Today is day 2 of my new job working at the Network Opoerations Center (NOC). I am still getting setup, there are quite a lot of apps and lot to know, so it will be a while till I'm comfortable, but so far so good. I think I will really enjoy it, and working at the central Office is pretty cool. I'm actually starting to somewhat understand the equipment there and how it talks to each other. It's pretty cool.

No more crazy IT manager to deal with. All the problems of that other place are now behind me! Like any other job, I'm sure this new one wont be perfect, but nothing can be worse than dealing with that other place and the crazy IT manager. Also the pay is better, and once I'm trained and able to work alone, I will be able to start shift work which will be a lot of fun. More time off! I will probably find a hobby of some sort to make more money on the side, or just expand on existing projects.

Friday, November 18, 2011

So I bought a playbook

It may not be the leading edge tablet, but at $300 off, making it $199, I could not resist for such a deal.

The only thing I'm kinda bumbed out about is that there are not many apps available yet, but I'm hoping more will be coded over time. I can always look at coding my own too, but for the meantime I'll wait and see what the app world will have to offer.

One thing I would love to see is an email client, especially one that can do imap. But with a web browser, anything is possible. I can always code a web client front end and put it on my server. A SSH and RDP client would be nice to see too. Though it does have VNC.

Also, Angry Birds would be nice to have on there. :P

But all and all, I don't regret the purchase. It's been a while since I've bought myself a toy. I've had to support these as well as other tablets at work, now I actually have one of my own to mess around with.

Friday, October 21, 2011

Learn from my fail ...and FML

When restoring a SQL database to a temp database so you can manually extract data, ensure that the backup job does not generate a database file that includes the original database name in it, causing it to restore to live instead.

Hopefully nobody got any uber rare artifacts today, because they are now gone.

Monday, October 17, 2011

What to do with an old power supply and ethernet cables?

The answer is: magnetize screwdrivers! Oh, and you'll want a piece of PVC pipe or something.

Remind me not to store my screwdrivers and hard drives in the same box. I kinda went overboard with this, and I have more Ethernet wire left. My goal is to reach around 5 ohms or so. Right now it's about 1.5 ohms, so without measuring I'm guessing that's pulling about 8 amps off the 12 volt rail. It gets hot enough if I leave it on too long it burns out. (this is coil #2!) It only takes about 10 seconds to charge a screwdriver to the point of being able to pickup a bunch of screws at once. As long as it can pickup one, it's usually good enough.

As a side note, they sure don't make stuff like they used to. This is what happened when I went to unplug a new Christmas light set:

Oops. Definitely a good idea that I'm installing a GFCI outlet for that.

Sunday, September 18, 2011

Hammer time!

This slab laughs at my pity attempts. I'm hoping to have weakened it enough to do the rest with the sledge hammer once I have energy again. There's a huge footing which will probably require some more serious equipment to take out.

The plan is to have this all gone by next year since I am getting the weeping does redone.

Before starting:


Day 2, done for now:

Tuesday, August 30, 2011

Gimme my new job!

I got accepted for a new position April 14 2011... it's now almost September 1st. The IT manager here is being more and more of an asshole with me latetly. Get me out of this place! The beauty of the new job is it wont be customer facing. Sooooo looking forward to it.

Today he starts freaking out because a user could not login to their mail, so I check into it, turns out they were part of a list to be deleted, so I restored it. He starts flipping out that I did not contact him, well... if he had used the proper process of putting in a ticket instead of just barging in here like a mental, then maybe I would of had a call back number. Then he starts flipping out at me for not putting the contact information in the account. woah woah there asshole, I'm not even the one that created that account, and it's not my fault HR does not provide the information half the time. DIAF. I so wish I could tell him off sometimes. He can be an asshole towards us, and we just have to take it up the ass. Sick of this. It's sad how people like him can have enough power to get away with this verbal abuse.

More on Hedge Clearing

This was done a while back, but thought I'd post. The stumps have been removed:

A bigger job than it seems, to do without machinery. Used an axe and shovel to rip em right out. Grass will eventually be grown there, but that will wait till next year after the weeping tiles are done.

Monday, August 29, 2011

Power outages: A server's worse nightmare

So you have a bunch of servers humming away, then suddenly, the power goes out. No problem, the UPS takes over, and when the battery starts getting low, the VMs are sent a ACPI shutdown, and once they are all shut down, the host is then sent an ACPI shutdown, and everything is gracefully shut down.

Later, the power comes back up. I personally like setting my machines to stay off, and I will manually turn them on. This prevents fast on/off if the power is repeatedly coming on and off and the battery is depleted. This is common when the cause of the power is a loose fuse in a pole as it may take a couple tries for the linemen to get it to click in properly... that long pole can't be easy to maneuver! Anyway, I come home once the power is fully restored, and I turn everything on.

This is where the problems begin. When electronic equipment is always on, at some point it's almost like it gets "used to it" and does not like to be shut down then turned on again. Guess the cooling of the components causes issues with the solder or what not. So the server boots up fine, I then start all the VMs. Not 5 minutes into booting up, BANG, one drive falls out of the raid 5 array. I check the serial number and my documentation to check which bay that drive in so I can go swap it with a replacement (I don't have a fancy setup where a light goes on when a drive fails). BANG! Another drive drops out. Game over, the raid array is toast.

After hours of playing around, I actually manage to get the raid array going again. Some data got corrupted and had to be restored from backups, but I did get it back without going 100% to backups. BANG a drive drops again. This time, for fun, I decide to insert it into another slot and let it rebuild. So far so good, but I can't trust this. My suspicion is the backplane, cable, or sata controller. I don't have time to do micro troubleshooting like this... I just want it to work, period. So I order two new backplanes (one is still backordered...) and 4 new drives (2 in there were already less than a week old). I can manage to get everything to work with just one backplane, and just one loose drive, so I go with that for now. So I slowly start removing the Hitachi drives and replace them with the new WD blacks letting it rebuild in between. No issues. I suspect maybe there is something up with those Hitachis so this is just a shot in the dark. Ironicly I had just finished RMAing 5 of them, which came in the mail the same day as the backplanes. But these particular drives were not showing any signs of issues so it was precautionary.

So that's done. Brand new backplane and drives. So far so good... then I start getting errors like this:

INFO: task pdflush:14763 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
pdflush D ffff8801dac8fa80 0 14763 2
ffff8800048c9da0 0000000000000046 ffff8800048c9d00 ffffffff8101686f
ffffffff8162a500 ffffffff8162a500 ffff88020f12adc0 ffff8801ff4496e0
ffff88020f12b108 000000020dc05dc6 ffff880028062570 ffff88020f12b108
Call Trace:
[] ? read_tsc+0xe/0x24
[] ? __dequeue_entity+0x61/0x6a
[] ? __switch_to+0x1b0/0x3e0
[] __down_read+0xa3/0xbd
[] down_read+0x2a/0x2e
[] sync_supers+0x4a/0xc4
[] wb_kupdate+0x35/0x119
[] pdflush+0x16e/0x231
[] ? wb_kupdate+0x0/0x119
[] ? pdflush+0x0/0x231
[] ? pdflush+0x0/0x231
[] kthread+0x49/0x76
[] child_rip+0xa/0x11
[] ? restore_args+0x0/0x30
[] ? kthread+0x0/0x76
[] ? child_rip+0x0/0x11

VERY bad. At least, it looks pretty bad. The task that hangs is random each time, and these errors can go on for pages. When it happens, all the VMs crash with IO errors.

Today I ordered a new controller, as that's the only thing that has not been changed short of the motherboard... with sockets changing every couple weeks it seems if I get a new mobo I'll need a new CPU as well, and may as well go with DDR3 and go from 8GB to 16GB. This server is really getting expensive.

Reproducing this error is near impossible as well, as I can give the hard drives a run for their money and they still perform fine, and it wont happen, and let it idle, and it may, or may not happen, for days. What a pain! It seems to always happen overnight, though there's a few instances where it's happened right in front of my eyes. Basically when it does, everything just locks up for a good minute or so.

This is why my next server will be prebuilt. Looking at a Supermicro or even Dell. Then again, even prebuilts can have their fair share of problems and parts are even harder to get. So it's a battle between building a whole new server, or getting a prebuilt, but either way, I think this thing is on it's last legs and in dire need of replacement. I am hoping the new controller will do the trick, however.

Thursday, July 28, 2011

Cleared some heges

I will be getting the foundation seal and weeping tiles redone next year, so I decided to start clearing the hedges for the excavator to pass through.

I even removed a few of the stumps, still need to do the others another time.

Also got rid of most of the patio stones, someone else was able to use them so they did not go to waste! Once the weeping tiles are done I will build a wooden deck here instead. Oh, and that existing deck slab has to go too, that should be some fun.

This stump should be fun to remove.

Removed the first few stumps, thankfully they're rather small.

Here's an unrelated Youtube video where you can see what it looked like before.

Wednesday, June 29, 2011

Crazy day

So yesterday was a deployment date for a program used by the entire organization, about 500 computers. The program itself had been tested by the IS group prior to me writing the install package... they should have given me the info and allowed me to create the install package, THEN test with THAT package, but whatever, they tend to do their own little thing then come rushing to us last minute to get something done.

So I had about two weeks to put together the install package. Enough time, but not really enough time to come up with a proper test nor to think up every possible scenario. I can test all I want in a VM, and the few machines not in use by users, but this environment is a HUGE MESS to begin with. We are not allowed to perform any type of updates or anything to the computers, and there are so many proprietary apps on select PCs it's sickening. Basically not one computer is the same so results vary on a per computer basis, when doing a deployment.

So yesterday things were quiet as most people probably did not reboot.

Today, it was like opening the floodgates of Hell. Helpdesk was getting flooded with calls because the program had this weird issue where when you open it, it just closes right away along with other problems. I was working on trying to figure out WTF it was doing this. Being a proprietary app itself, there was no documentation or anything to refer to. Obviously, closed source too, so can't look there. Basically not much help available. It's not like you can Google for an answer with an app that is this proprietary.

We later on figured out that the app requires the folder to be world writable so I wrote a separate script that I can push to change the NTFS permissions with CACLS.

Found a page with some undocumented features which allows to set group permissions (the documentation only mentions users)

Being under pressure, my first attempt at writing a script to push a fix had a typo which caused bigger problems, but I quickly found and fixed it, and within an hour or so everything was golden.

The IT manager, who is a real asshole, barged in with his typical asshole look on his face and wanted an explanation for everything. That guy will ensure to take a bad day, and turn it into your worse nightmare.

Tuesday, June 21, 2011

Weekend yard work and stuff

Was nice this weekend, got some stuff done, such as trimming my pine tree...

Also used a jackhammer for my first time... Did not get much time so decided to rent it for a week when I'm off, will be cheaper as this will probably take a couple days.

Oh Yeahhhhhhh

Tuesday, June 14, 2011

The solution to not enough power outlets

Ever find yourself not having enough power outlets? With a UPS setup you want certain things protected (such as the pc and monitors) and certain things not (such as a printer). That means you have at least two power bars. Some things such as your speakers or phone charger take up like 3 outlets. Next thing you know you are daisy chaining power bars.

What a mess!

The solution? A DIY 16 outlet power bar rated for a total of 30 amps! Nevermind the fact that you can run a mini data center off this, it's just really cool looking and you should hopefully not run out of outlets.

More details here

Randomness about Linux

I've always liked Linux for it's superior robustness especially when it comes to automation. Compared to Windows it is so easy to automate tasks as everything is command line friendly and usually natively command line. GUI stuff simply calls command line applets. There are lot of high end tools such as rsync for backups, and bash scripting for advanced conditional based automated scripting, which is far superior than Windows batch files. Powershell is an improvement by Microsoft, but it is an add-on, so to me that does not count. You can't write a powershell script, send it to a friend, and have them execute it, or use it as a login script. The PC wont have the program installed.

Today, I decided to convert my Open Filer box to a straight Linux box, to manage the raids manually. While Open Filer is pretty good, I found it lacked certain things such as being able to grow raid 5 arrays from the GUI. I found myself in the command line half the time doing stuff the GUI could not do. Figured I may as well do it all from command line. I may also use the box for something else. Debating on DHCP. Speaking of which, ever do a HA setup in Windows for DHCP? Pure nightmare and just overall tedious. Linux is so much easier.

This is another great thing about Linux. Being open source, anyone can take it, modify it (such as make a distribution like Open Filer) and redistribute it. Heck if I wanted to I could probably add that functionality in Open Filer and redistribute it. There's no copyright, no patents, none of that crap. You are free to do what you want and nobody is going to penalize you for it. This opens tons of doors when it comes to custom solid solutions.

Some people are highly against Linux in corporate environments "Because it's free and there's no support" (which is false, plenty of distros that come with optional support). Yet, lot of corporate systems are actually based on it! VMware ESX/i just as an example. Lot of embedded devices are also based on it. The reason they do this is because they can. Try that with Windows. It will be cumbersome, and have licensing issues.

Linux FTW.

Saturday, April 9, 2011

Got cable!

Been living in this house for almost 2 years, finally decided to get cable. Also installed appropriate data jacks and an electrical outlet behind the TV cabinet. I have a HTPC which streams videos from my main server as well through gigabit ethernet. Overall it's quite a great setup.

The TV setup:

I need more remotes!


Thursday, February 24, 2011

Moved server hardware to new case and racked it

Today I took part of the day off to get into my 4 day weekend, given I had 2 days and 3 hours left in holidays, so I decided it would be a nice time to convert my main server to rackmount.

Was quite the job, but it went rather smooth, and powered on first shot. The only mistake I did was plug the power LED backwards.

It's quite a mess in there, but it works, and that's what counts. I may add another rear fan though as there is only one right now. I taped up lot of the back vents to force air to go through the hard drives.

And finally, racked up:

Wednesday, February 23, 2011

Rackmount Server Conversion Progress

I finally got my server rails today! Going to convert my tower server to rackmount. I think this is the only thing that kept me sane today, knowing I had a new toy come in.

That server case was ordered originally but I thought it came with rails. Paid a bit over 100 bucks for the server and rails together. I will probably tweak the air vents a bit to maximize air flow. Idealy I should have ordered another fan.

Wednesday, February 9, 2011

Fired up the SAN and Open Filer

Installed Open Filer on the Atom SuperMicro server I bought, hooked up the fiber to two enclosures and fired it up. Currently making two raid 10 arrays, with a total of 4 volumes. 1 iSCSI and 1 EXT3 volume per array. I never made a raid 10 before, figured now that I have all these drives, it's a good time to do so.

I am not sure yet if I will keep the switch there, as it is a much simpler setup to just go straight to the enclosure and use it as a DAS. I can always get another server like the one I have in the future, and do a high availability Open Filer setup, but that's beyond the scope of this project. I will not even be using this for production, it's more or less a fun toy.

The two arrays processing:

The nice thing with Open Filer is that it uses the Linux MDADM raid which is fairly standard and very easy to use even outside of Open Filer. Should the enclosure fail, or the fiber card fail, or any of this hardware otherwise stop working, I simply need to put those drives in a new enclosure, present it to any Linux box, and the data will be available again. It's really that easy, this is why I really prefer open source over proprietary stuff.

Once the raids are done rebuilding, I will play around with iSCSI and see how that goes.

Thursday, January 27, 2011

Bought mini Atom Server

Got my shares, so I decided to buy myself a Supermicro Atom server. I installed a 500GB hard drive and a Qlogic Fiber HBA which will be to use with my fiber channel SAN. Never played with that equipment but I am hoping I can just make it so the server sees each hdd individually, from there I will use Open Filer to create a SAN. I want to avoid using the proprietary SAN's software as this thing is old and could possibly crap out any time. If I stick to something semi standard recovery will be easier. I've also been wanting to play with Open Filer, so this will be a great opportunity.

(don't mind the mess on there, lot of random stuff lol)

Saturday, January 15, 2011

Server migrations are tedious

Almost done migrating my online linux server, so many little things to reconfigure and setup, it's a pain! One of these days I want to write an application to automate all of these tasks and a server move will be a one click operation!

One of these things I forgot was to change the IP on the DNS record for the blog images. They should now show up within 10 minutes...

Next and one of the final tasks is to move the UO game server over. Need to setup a windows VM for that since I have yet to get my version of RunUO to work in Mono. Need to rewrite the code so it is compatible. HUGE task.

Tuesday, January 11, 2011

Crawlspace lighting

I have a crawlspace under my garage which is accessible from the main crawlspace under the lower half of the house. It has no lighting and I occasionally go in there to vacuum up all the spider webs. It is very tedious as I need to bring a lamp with me and the wires get all tangled up along with the hose of the vacuum. So I decided it's time to install lighting. I also will be hiring a spray foam company to insulate it so I'm actually doing them a favor by having the lights there.

When I first bought the house, the crawlspace looked like this:

This is a pic of when I first bought the house. I had went in there at one point after I bought it, so this time it was not as bad.

I have a phobia of spider webs. Not spiders, but spider webs. They just look so creepy, and I screem like a little girl if I come into contact with one. They just feel so eerie. Meanwhile, I don't mind picking up spiders or squishing them with my finger. Figure that one out.

So after slowly moving through that area with the shop vac I cleaned it all of webs and proceeded with the install.

Started off by installing some boards to mount the electrical on, just easier that way.

This crack and a few others made it a little scary working under there. In rebar I trust.

There used to be one switch here for the main crawlspace (below). Decided to feed the new crawlspace lights off the same circuit and add it's own switch.

Added a conduit to go through the wall (garage crawlspace on other side). Eventually I will add studs to this wall and drywall it, so I will secure the wire at that point. I will probably also spray some foam in that conduit just to seal it. Minor details.

Lights installed. 2 bulbs on this side of the beam...

...And two more on the other side. Those are 26w CFLS which are equivalent to 100w incad.

Since I had previously cleaned the spider webs and I could freely move around in there without catching one in the face, I also added an ounce of cyonara 9.7 in a gallon of water and sprayed all surfaces. That should greatly cut down on spiders and other bugs.

More pics