Red's Blog

It's official, I finally switched to Windows 7

2012-05-02T22:11:00.001-04:00

I completed building my new computer today (was off) and installed windows 7.

There is still tons of stuff to install and configure, but things are falling together. Lot of the common apps like Firefox and Thunderbird are fairly easy as I can just copy over the profile and it takes it.

Overall I'm quite happy with 7. Some stuff is different, but it's not that hard to find things, and the annoying stuff (like UAC) is easy to turn off.

I switched it to Windows classic, then decided to switch it back, the default is kinda growing on me, I think I'm actually going to stick with it. The computers I had tried it on before were slow so it was really choppy, but on this machine that's the least of my worries.

I'm only using about 2GB/12GB of ram. I freaked a bit when I saw that 80GB of my 120GB SSD was used, turns out it was the page file, disabled that, with 12GB of ram I wont need it, and I would rather put it on a spindle (or at least separate SSD) drive if I did end up using a page file.

I will post pics of the actual PC later. I'm still waiting for my 1000w power supply to come in. I gave up waiting and bought a 600w to get me by as it was the only missing part. So I did not hookup the SLI cable just yet, I'll wait.

After 5+ years since my last build I have to say lot of things changed hardware wise.

1. Everything is so HUGE. The video cards, heat sink, even the ram now has huge heat sinks on it.

2. It's still hard to find a case that has proper ventilation in mind. I still found one though. Decided to go for a full positive pressure setup, all intakes, no exhausts. What would be exhaust ports are left open obviously, and the air naturally goes out of them. That way the combined air flow of the fans is higher than if I had some intake and some exhaust.

3. Instructions don't get very in depth. It took me a while to figure out how to install the cpu and the heat sink. I had a feeling I knew how as it's not very hard, but when you're dealing with over $500 of equipment (the cpu and motherboard specificly) you don't want to "think" you are doing it right, you want to "know". After a bit of research I confirmed I was doing it right, so I went ahead, no issues.

4. Included accessories: there is much less now. CPUs used to always come with a heatsink/fan, not anymore! Found that out the hard way. Also I did not even get an Intel Inside sticker. :( I want my sticker! It makes the computer go faster.

A bit more on point 1:

When I got the heat sink on, I found out the hard way that I could only put 2 sticks of ram not 3 or 4. At first I was going to settle with 8GB but the more I got thinking, the more I realized I can't just settle for 8, when my 5 year old server has 8. I want to go a step ahead of things here! I analyzed the situation and tried to figure out what I can do. Turns out if I turn the heat sink around I can fit the ram. Cool! It was VERY close to not fitting that way because one of the video card though. Speaking of which, I was not able to add the sound card, because there are only 2 of the 16x slots that actually run at 16x. One of which interfere with the only PCI slot. So I decided to leave out the sound card for now. If I find the onboard is not good enough (they tend to be bad when gaming) I will get a USB one. I do have an available PCI-e 1x slot but I may want to put a network card in there. I like having two network cards so I can connect to my external network directly if I have to, without disconnecting from my main.

At the end, everything went fairly smooth. Best part: no DOA parts! I think this is my first build where every part actually works! First time I run into issues where stuff does not fit, but it turned out to not be so bad. Only left with no sound card.

I'm still installing and configuring everything, I can't wait to actually game on this! I also need to find some kind of widget to monitor the temperatures, I always like having that visible.

New PC on it's way!

2012-04-23T20:08:00.000-04:00

Can't wait to put this beast together! It did not ship yet, I'm hoping it ships tomorrow! If I can get it by Friday I'll be happy so I can put it together over the weekend. Hopefully there wont be any DOA parts, I tend to have bad luck with that.

More HVAC system pics

2012-03-13T23:05:00.001-04:00

Some more pics:

Mounted control board and soldered in all the wires. Idealy I would have loved to make it terminal blocks instead but never even thought of ordering some direct mount ones when I did my order for the other stuff. I will try to find a piece of transparent plastic to use as the cover.

Can now control furnace with my playbook or any wifi device

I thought this was rather genius...

Conveniently this is a great location for an emergency light, or just a light to look inside the closet beside it.

Server room sensor (this "room" is basically my basement at this point)

Return duct sensor

Supply duct sensor

Top view of rack where the control board sits on a shelf and the DIN rail is mounted on top. The actual server is closer to the middle and not shown, it's a little 1U SuperMicro server.

And lastly, here's a pic of the front end. I will probably tweak it more over time and make it look nicer.

We're int he middle of a heat wave right now so wont get to use this to it's full extent for another couple days, but in a way it's a good thing as I need to do a simulation test for a few days anyway before letting it activate the relays. Though I'm 99% sure I sorted out the issue now.

I also need to hookup the outside sensor but I'll wait till summer for that.

I always say that I quit, but never do

2012-03-13T21:48:00.003-04:00

When I made my last post I was about ready to quit this. The control board was acting stupid, I was unable to get any support for it, and I was getting pissed off considering it worked before and suddenly would not work after a server reboot. I have been pulling my hair out all week.

Well, I think I may have figured out a fix. I still don't understand WHY it does this and I still think it's some kind of bug, but by looking at different online examples of serial connectivity and copying code around, I changed my connect function to this:


bool SerialConnector::Open()
{
 if(m_Path=="")
 {
  ScreenOutput("No path specified");
  return false;
 }


 
 //init stuff
 
 struct termios tio;
 //struct termios stdio;
 //int tty_fd;
 //fd_set rdset;


 //memset(&stdio,0,sizeof(stdio));
 //stdio.c_iflag=0;
 //stdio.c_oflag=0;
 //stdio.c_cflag=0;
 //stdio.c_lflag=0;
 //stdio.c_cc[VMIN]=1;
 //stdio.c_cc[VTIME]=0;
 //tcsetattr(STDOUT_FILENO,TCSANOW,&stdio);
 //tcsetattr(STDOUT_FILENO,TCSAFLUSH,&stdio);
 //fcntl(STDIN_FILENO, F_SETFL, O_NONBLOCK);       // make the reads non-blocking 


 memset(&tio,0,sizeof(tio));
 tio.c_iflag=0;
 tio.c_oflag=0;
 tio.c_cflag=CS8|CREAD|CLOCAL;           // 8n1, see termios.h for more information
 tio.c_lflag=0;
 tio.c_cc[VMIN]=1;
 tio.c_cc[VTIME]=5;


 //connect
 m_fd=open(m_Path.c_str(), O_RDWR | O_NONBLOCK); 
 
 
 
    cfsetospeed(&tio,B115200);            // 115200 baud
    cfsetispeed(&tio,B115200);            // 115200 baud 

 //this is required, it seems that it only needs to be called once per server reboot. It sets some kind of global thing that prevents the serial device (in my test: uk1104 from canakit) to go insane and spam junk
 tcsetattr(m_fd,TCSANOW,&tio);
 
 if (m_fd == -1 )
 {
  ScreenOutput("Unable to connect");
  m_conactive=false;
  return false;
 }

 m_conactive=true;
 
 return true;
}

I want to thank the author of this page:

https://en.wikibooks.org/wiki/Serial_Programming/Serial_Linux

The example is straight to the point line by line and easy to follow. Most of the examples I found were terrible to figure out.

I don't really know what every single line of code does, but the key lines are these:





 memset(&tio,0,sizeof(tio));
 tio.c_iflag=0;
 tio.c_oflag=0;
 tio.c_cflag=CS8|CREAD|CLOCAL;           // 8n1, see termios.h for more information
 tio.c_lflag=0;
 tio.c_cc[VMIN]=1;
 tio.c_cc[VTIME]=5;

 
 
 
    cfsetospeed(&tio,B115200);            // 115200 baud
    cfsetispeed(&tio,B115200);            // 115200 baud

In that order. I'm guesssing setting the baud rate may possibly play a role, but the code above also does. The connect function is the same as before.

It seems to me some of these parameters stick even after the program is done executing. I discovered that when I ran an example stand alone, and ran my program, my program worked. If I rebooted, my program stopped working. Oddly, running picocom did not actually set whatever global flags were being set, at least not permanently.

Either way, after several server reboots and several resets of the USB device it seems to work. Though, I may be calling it too early. I will let it run overnight before I take the app out of simulation mode, (basically this mode just ignores relay calls).

After tomorrow I will be off for 2 days, so I will run it live and see how it works out.

I really hope this works out on a long term basis, as I'm looking forward to actually putting this to good use and no longer need to keep messing with the thermostat programming day to day to accommodate my different shifts. This app is set and forget, which is awesome. It's why I did it. Given most people work standard shifts, consumer thermostats are designed for that, so I needed something to accommodate shift work.

Once this is proven, I will probably make up a full blown article explaining everything I did. Man, I'm so glad I got this working though, this spam thing was really starting to burn me.

Well this was a waste of time

2012-03-13T19:57:00.004-04:00

The control board decided to crap out and now it refuses to work. IT did this before but resetting it would fix the problem. Now it wont work period. As soon as I try to write anything to it, it just starts spitting out bullshit.



::



::



::



::R

ERROR : R



:::

ERROR : :



:::

ERROR : :



::



::



::



::::

ERROR : ::



::



:::

ERROR : :



::E

ERROR : E



::



::



::



::



::



::



::



::



::



::ER

ERROR : ER



::



::::

ERROR

Canakit has not replied to any of my emails on this issue, so it looks like I'm on my own. I need to find another board that can interface with the Dallas thermal sensors.

What's screwed up is if I use picocom, it works fine, but the minute I try to use echo/cat or my own code, it does this crap. I even copied the code from picocom into my own code, and it STILL does it. I'm at a loss. This project was just too good to be true. I was SO close too. :( I had to go reboot that server... but guess it's better now, than down the line when a relay is turned on and I'm not home...

So I just did the final hookup of my hvac system

2012-03-10T19:40:00.001-05:00

I just officially started the furnace using my playbook. So far the system is working. I've been testing it for a long time using a light bulb and sensor in a bucket so it's been fairly proven. Today I now did the hookups to the furnace. I wired it so my old thermostat can serve as a backup, it's in parallel.

I just have one more sensor to hookup, some cleaning up to do and a few minor final steps, and the system will be officially completed.

More progress on HVAC control system

2012-03-06T03:03:00.001-05:00

Started pulling wire today. Also decided to change up a bit the way I run wire around the house. Instead of just hanging in screw eyes and tie wraping any new cabling on the existing cabling that's passed through the screw eye (normally only one can fit) I decided that I will run a metal wire in the screw eye, pass the first wire into the screw eye, but use velcro straps to hold additional wires. The metal wire also acts as something to strap wire to between joists to prevent hang. So I did that today as well.

Wired the first sensor into the thermostat, so it's more or less hidden. I decided to make the sensor part itself stick out on the outside, for a more accurate temp reading. The thermostat will remain as a backup and be wired in parallel for the heat only.

After I took that pic I realized I wired it wrong, The orange and brown/white wires are mixed up. Mostly mentioning this in case I come back and look at this in the future.... lol. I just made my own "standard" that I will follow for each sensor. Will make future troubleshooting easier. The sensors have 3 wires, so I just assigned each one to a color.

And here's some runs (some of those are existing Ethernet runs).

Crawlspace data run, the one going to the far end is actually existing Ethernet, the one going behind the light is for the sensor.

A run to the outside sensor. I will leave it coiled up at the end till summer. I don't really want to work outside in this weather, and I doubt any glue/cement I use would cure properly anyway. My plan is to have a small pipe go outside and have a down elbow with a cap and a couple holes in it. The sensor will reside at the tip of the pipe and the rest will be filled with insulation and caulked in. Will provide weather proofing for the sensor. I specifically chose a north wall which should provide an accurate reading. Most of these sensors are just for extra informational readings, more than anything. Though they will be used in some logic as I improve the control application.

Din rail with the terminal blocks. They are all labeled to make it easy to identify each conductor.

Hoping to more or less finish this up tomorrow, if not, next week. I'm off tomorrow but then start work again the next day. I still have one more wire run to do for the upstairs sensor. I bought a LED wallmount light which I'll be using to mask the sensor into, and the light itself may come in handy in the event of a power outage or what not.

Almost done HVAC Control app

2012-02-21T00:26:00.002-05:00

More or less done the back end and front end now. There's still more testing and tweaking to do, but so far it's looking good.

I decided to omit the algorithm to determine when the furnace should actually start based on a scheduled time. Figured I should ensure that what I have works and then throw that feature in afterwards.

I'll finish up the tweaking tomorrow and then next time I'm off I'll start on the wiring. I ordered 4 more sensors so they should hopefully arrive by then.

I'll have one in the living room, upstairs, server room, furnace return duct, furnace supply duct and one outside. I'm debating on instead of putting one outside, I can just use data from the weather channel. Though real time actual temp would be kind of cool.

This is what the main interface looks like:

The living room sensor is not actually installed so that reading is from the sensor just sitting in the server room crocodile clipped to the circuit board. It looks good on my Playbook too and is easily usable. I will probably do a "dry run" test for a couple weeks before I do the final hookup to the furnace. I'm paranoid about it randomly dropping the serial/usb connection with the relay turned on. Could be disastrous if it happens when I'm not home. Since I'm not using the USB extender now it has not dropped though, so think I'm good.

Ever see an icicle grow from bottom up?

2012-02-14T19:30:00.002-05:00

Here's one!

Been growing for a couple months at least.

hvac control web front end

2012-02-12T19:30:00.004-05:00

I started on the front end for the hvac control. In my mind I had really fancy ajax stuff with graphs and whole 9 yards, but then I realized I don't even know ajax so I think I will keep it simple for now, and might add on to it later, and also make it look more pretty.

This is the programming section. A program is basically a set of "rules" that will be followed for a single day. Each day has a program assigned to it. It's very similar to a programmable thermostat where you set a temp for a certain time and it will hold it till the next program starts.

There is a calendar where the program is assigned to an actual day. Note the lack of "weekly" schedule setting. I originally was going to make it so I can set a schedule on a per week basis then go back to override it but it got too complicated for nothing. I work shifts so I'm always going to be going to change it anyway. I will make the back end default to a certain program if no program is specified. Basically it will default to the vacation one.

I did not do the dashboard yet, but that will basically just be a display of the current temp, the program that is active, the temp goal, and an option to do a temporary override without changing the program. Basically an override will be a specific temp at a specific time and a specific duration. This will take priority over the program.

One thing you will notice is that instead of specifying a temp, it's a temp range. This will be more flexible and more efficient as it will avoid short cycling. I will tweak that as I go to get a balance of comfort and efficiency.

After that, I just need to do the back end, which should not be too hard as I already wrote the control class for the relay/sensor board. The back end will basically just be a looped app that checks the sensors, and checks the program, and updates/reads the sql database.

In the future I may make it plot temp graphs and stuff. There will also be some logic such as starting the furnace earlier based on the outside temp and indoor temp. I can probably figure out a formula for that, otherwise I can just make it learn over time.

Din Rail installed

2012-01-31T22:03:00.002-05:00

Been working mostly on the software portion, but as a side task I did cut and install the DIN rail to my server rack. I think it should look pretty slick once I have the connections in place. I don't know how common it is to put a DIN rail on a server rack, but hey, it works!

I think my serial class is more or less proven for the temp sensing and relay control, so I will be starting on the actual thermostat app next. I'm hoping to get something rough done within a week or so, I can add the more fancy eco algorithm stuff later down the road.

DIN rails, wire and DIN terminal connectors

2012-01-27T17:53:00.002-05:00

This stuff came in today, which will be used for this hvac project, and possibly other future stuff.

The DIN rail will be screwed into the server rack and treated like a patch panel for misc sensor/electrical connectivity. Basically anything that does not really have a plug type. For now I'll only be controlling the furnace, and I'll only have 2 thermal sensors. One where the existing thermostat is, and the other I'm not sure yet. Either the server room, or outside, but winter is not an ideal time to be putting holes in outside walls. The circuit board can handle 6 sensors in total so I may put the second sensor indoors and wait till summer to order more and put one outside.

Once the outside temperature sensor is installed my program will be able to use that to better control the furnace start times based on the schedule. For example the app will know to start the furnace 1 hour before a schedule based on the outside and indoor temp.

HVAC temp sensor

2012-01-20T23:52:00.003-05:00

Using a makeshift breadboard to wire up the resistor and temp sensor (aka: cut out a piece of plastic from HDD tray inserts)

The unmarked wire is for ground, the one with one line goes to the DQ pin of the sensor and will go to the channel pin of the control module. The wire with 2 markings is the VDD pin of the sensor and will go to the +5v line of the control module. I will be using cat6 to wire this to the module, so probably one pair for DQ and VDD and a single wire for ground. I can probably technically use the common wire of the thermostat as ground, but I rather use the one fed from the control module to be on the safe side.

I will try to shove this behind the existing thermostat which will save me from figuring how to put this on the wall and hide it.

There is also another sensor that will go outside, but not sure how I want to do that yet, and considering it's like -30 out there, I rather wait till spring to do any type of outside work. Chances are what I'll do is mount it on the house in some kind of small plastic box. Might even put it inside the phone nid.

The outside sensor will be used to help the system better calculate start times and run times to increase efficiency. For example, on a -30 day I might want the furnace to start earlier than a -5 day. the current indoor temp will also play a role in that calculation. The program will learn the heat gain/loss and then based on these values should be rather accurate in knowing when to start the heat.

There will also be other features such as over running past the set temp to avoid short cycling. So on a -30 day it may be better off to run the heat 2 degrees past what it's set to, and let it drop 3 degrees before starting again. The less short cycling, the more efficient things will be. Still lot of things to think about. At first the program will simply be an advanced temp scheduling program and I'll add more features over time. Being able to set the temp or change schedules from the office, or my playbook will be pretty cool.

21st century HVAC control

2012-01-16T22:18:00.002-05:00

I just got this in the mail today:

It's a 4 port relay controller with 6 input sensors. I also got 2 temp sensors, but really I should have got 6 from the get go and be done. Might put in another order once I confirm it works as I want.

My goal is to hook this up to my mini atom server which I will dedicate as a environmental control system. The relays will be used to control the heat and the temp sensors as a thermostat. I will code a web based app to control it. I plan to add lot of intelligence to it such as know how early to start the furnace based on the outside temperature. Once this is done I'll be able to control the furnace from work. I will be starting shiftwork soon, too, so being able to setup more complex schedules than the typical weekly/weekend settings will be great. I will also have a function to quickly override a schedule for a single day. Sometimes I go for lunch at 12:00 other times I go for lunch at 1:00 etc we just kinda play it by ear. So when I determined when I go for lunch then I'll set it to start so it's warm when I get home. Should be a fun toy to play with, and it will be practical.

I now need to decide how to physically set it up. How I wire it, etc... to make it as clean as possible. I might mount the board right into the server somehow, then have a terminal strip on the back of the server to make the thermostat and sensor connections. For now I'll have one sensor where my current thermostat is (I'll just shove it right in it basically) and the current thermostat will be used simply as a backup. It will be wired in parallel.

Later on I might add more sensors, like one on the upper floor, one in the basement, one in what will be the server room, maybe one in the garage.

I will only need 3 relays, so the 4th will probably be used for some misc purpose. Maybe the hvac for the server room itself once I build that room.

Now if I could add voice recognition to that server and have sensors around the house, that would be awesome. The server is called HAL9000. :D

Finally Started my new job!

2011-12-02T18:41:00.002-05:00

Today is day 2 of my new job working at the Network Opoerations Center (NOC). I am still getting setup, there are quite a lot of apps and lot to know, so it will be a while till I'm comfortable, but so far so good. I think I will really enjoy it, and working at the central Office is pretty cool. I'm actually starting to somewhat understand the equipment there and how it talks to each other. It's pretty cool.

No more crazy IT manager to deal with. All the problems of that other place are now behind me! Like any other job, I'm sure this new one wont be perfect, but nothing can be worse than dealing with that other place and the crazy IT manager. Also the pay is better, and once I'm trained and able to work alone, I will be able to start shift work which will be a lot of fun. More time off! I will probably find a hobby of some sort to make more money on the side, or just expand on existing projects.

So I bought a playbook

2011-11-18T18:34:00.003-05:00

It may not be the leading edge tablet, but at $300 off, making it $199, I could not resist for such a deal.

The only thing I'm kinda bumbed out about is that there are not many apps available yet, but I'm hoping more will be coded over time. I can always look at coding my own too, but for the meantime I'll wait and see what the app world will have to offer.

One thing I would love to see is an email client, especially one that can do imap. But with a web browser, anything is possible. I can always code a web client front end and put it on my server. A SSH and RDP client would be nice to see too. Though it does have VNC.

Also, Angry Birds would be nice to have on there. :P

But all and all, I don't regret the purchase. It's been a while since I've bought myself a toy. I've had to support these as well as other tablets at work, now I actually have one of my own to mess around with.

Learn from my fail ...and FML

2011-10-21T23:23:00.003-04:00

When restoring a SQL database to a temp database so you can manually extract data, ensure that the backup job does not generate a database file that includes the original database name in it, causing it to restore to live instead.

Hopefully nobody got any uber rare artifacts today, because they are now gone.

What to do with an old power supply and ethernet cables?

2011-10-17T19:21:00.002-04:00

The answer is: magnetize screwdrivers! Oh, and you'll want a piece of PVC pipe or something.

Remind me not to store my screwdrivers and hard drives in the same box. I kinda went overboard with this, and I have more Ethernet wire left. My goal is to reach around 5 ohms or so. Right now it's about 1.5 ohms, so without measuring I'm guessing that's pulling about 8 amps off the 12 volt rail. It gets hot enough if I leave it on too long it burns out. (this is coil #2!) It only takes about 10 seconds to charge a screwdriver to the point of being able to pickup a bunch of screws at once. As long as it can pickup one, it's usually good enough.

As a side note, they sure don't make stuff like they used to. This is what happened when I went to unplug a new Christmas light set:

Oops. Definitely a good idea that I'm installing a GFCI outlet for that.

Hammer time!

2011-09-18T18:42:00.002-04:00

This slab laughs at my pity attempts. I'm hoping to have weakened it enough to do the rest with the sledge hammer once I have energy again. There's a huge footing which will probably require some more serious equipment to take out.

The plan is to have this all gone by next year since I am getting the weeping does redone.

Before starting:

During:

Day 2, done for now:

Gimme my new job!

2011-08-30T16:47:00.003-04:00

I got accepted for a new position April 14 2011... it's now almost September 1st. The IT manager here is being more and more of an asshole with me latetly. Get me out of this place! The beauty of the new job is it wont be customer facing. Sooooo looking forward to it.

Today he starts freaking out because a user could not login to their mail, so I check into it, turns out they were part of a list to be deleted, so I restored it. He starts flipping out that I did not contact him, well... if he had used the proper process of putting in a ticket instead of just barging in here like a mental, then maybe I would of had a call back number. Then he starts flipping out at me for not putting the contact information in the account. woah woah there asshole, I'm not even the one that created that account, and it's not my fault HR does not provide the information half the time. DIAF. I so wish I could tell him off sometimes. He can be an asshole towards us, and we just have to take it up the ass. Sick of this. It's sad how people like him can have enough power to get away with this verbal abuse.

More on Hedge Clearing

2011-08-30T16:43:00.003-04:00

This was done a while back, but thought I'd post. The stumps have been removed:

A bigger job than it seems, to do without machinery. Used an axe and shovel to rip em right out. Grass will eventually be grown there, but that will wait till next year after the weeping tiles are done.

Power outages: A server's worse nightmare

2011-08-29T21:46:00.002-04:00

So you have a bunch of servers humming away, then suddenly, the power goes out. No problem, the UPS takes over, and when the battery starts getting low, the VMs are sent a ACPI shutdown, and once they are all shut down, the host is then sent an ACPI shutdown, and everything is gracefully shut down.

Later, the power comes back up. I personally like setting my machines to stay off, and I will manually turn them on. This prevents fast on/off if the power is repeatedly coming on and off and the battery is depleted. This is common when the cause of the power is a loose fuse in a pole as it may take a couple tries for the linemen to get it to click in properly... that long pole can't be easy to maneuver! Anyway, I come home once the power is fully restored, and I turn everything on.

This is where the problems begin. When electronic equipment is always on, at some point it's almost like it gets "used to it" and does not like to be shut down then turned on again. Guess the cooling of the components causes issues with the solder or what not. So the server boots up fine, I then start all the VMs. Not 5 minutes into booting up, BANG, one drive falls out of the raid 5 array. I check the serial number and my documentation to check which bay that drive in so I can go swap it with a replacement (I don't have a fancy setup where a light goes on when a drive fails). BANG! Another drive drops out. Game over, the raid array is toast.

After hours of playing around, I actually manage to get the raid array going again. Some data got corrupted and had to be restored from backups, but I did get it back without going 100% to backups. BANG a drive drops again. This time, for fun, I decide to insert it into another slot and let it rebuild. So far so good, but I can't trust this. My suspicion is the backplane, cable, or sata controller. I don't have time to do micro troubleshooting like this... I just want it to work, period. So I order two new backplanes (one is still backordered...) and 4 new drives (2 in there were already less than a week old). I can manage to get everything to work with just one backplane, and just one loose drive, so I go with that for now. So I slowly start removing the Hitachi drives and replace them with the new WD blacks letting it rebuild in between. No issues. I suspect maybe there is something up with those Hitachis so this is just a shot in the dark. Ironicly I had just finished RMAing 5 of them, which came in the mail the same day as the backplanes. But these particular drives were not showing any signs of issues so it was precautionary.

So that's done. Brand new backplane and drives. So far so good... then I start getting errors like this:


INFO: task pdflush:14763 blocked for more than 120 seconds.

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

pdflush       D ffff8801dac8fa80     0 14763      2

 ffff8800048c9da0 0000000000000046 ffff8800048c9d00 ffffffff8101686f

 ffffffff8162a500 ffffffff8162a500 ffff88020f12adc0 ffff8801ff4496e0

 ffff88020f12b108 000000020dc05dc6 ffff880028062570 ffff88020f12b108

Call Trace:

 [] ? read_tsc+0xe/0x24

 [] ? __dequeue_entity+0x61/0x6a

 [] ? __switch_to+0x1b0/0x3e0

 [] __down_read+0xa3/0xbd

 [] down_read+0x2a/0x2e

 [] sync_supers+0x4a/0xc4

 [] wb_kupdate+0x35/0x119

 [] pdflush+0x16e/0x231

 [] ? wb_kupdate+0x0/0x119

 [] ? pdflush+0x0/0x231

 [] ? pdflush+0x0/0x231

 [] kthread+0x49/0x76

 [] child_rip+0xa/0x11

 [] ? restore_args+0x0/0x30

 [] ? kthread+0x0/0x76

 [] ? child_rip+0x0/0x11

VERY bad. At least, it looks pretty bad. The task that hangs is random each time, and these errors can go on for pages. When it happens, all the VMs crash with IO errors.

Today I ordered a new controller, as that's the only thing that has not been changed short of the motherboard... with sockets changing every couple weeks it seems if I get a new mobo I'll need a new CPU as well, and may as well go with DDR3 and go from 8GB to 16GB. This server is really getting expensive.

Reproducing this error is near impossible as well, as I can give the hard drives a run for their money and they still perform fine, and it wont happen, and let it idle, and it may, or may not happen, for days. What a pain! It seems to always happen overnight, though there's a few instances where it's happened right in front of my eyes. Basically when it does, everything just locks up for a good minute or so.

This is why my next server will be prebuilt. Looking at a Supermicro or even Dell. Then again, even prebuilts can have their fair share of problems and parts are even harder to get. So it's a battle between building a whole new server, or getting a prebuilt, but either way, I think this thing is on it's last legs and in dire need of replacement. I am hoping the new controller will do the trick, however.

Cleared some heges

2011-07-28T20:41:00.004-04:00

I will be getting the foundation seal and weeping tiles redone next year, so I decided to start clearing the hedges for the excavator to pass through.

I even removed a few of the stumps, still need to do the others another time.

Also got rid of most of the patio stones, someone else was able to use them so they did not go to waste! Once the weeping tiles are done I will build a wooden deck here instead. Oh, and that existing deck slab has to go too, that should be some fun.

This stump should be fun to remove.

Removed the first few stumps, thankfully they're rather small.

Here's an unrelated Youtube video where you can see what it looked like before.

Crazy day

2011-06-29T22:48:00.002-04:00

So yesterday was a deployment date for a program used by the entire organization, about 500 computers. The program itself had been tested by the IS group prior to me writing the install package... they should have given me the info and allowed me to create the install package, THEN test with THAT package, but whatever, they tend to do their own little thing then come rushing to us last minute to get something done.

So I had about two weeks to put together the install package. Enough time, but not really enough time to come up with a proper test nor to think up every possible scenario. I can test all I want in a VM, and the few machines not in use by users, but this environment is a HUGE MESS to begin with. We are not allowed to perform any type of updates or anything to the computers, and there are so many proprietary apps on select PCs it's sickening. Basically not one computer is the same so results vary on a per computer basis, when doing a deployment.

So yesterday things were quiet as most people probably did not reboot.

Today, it was like opening the floodgates of Hell. Helpdesk was getting flooded with calls because the program had this weird issue where when you open it, it just closes right away along with other problems. I was working on trying to figure out WTF it was doing this. Being a proprietary app itself, there was no documentation or anything to refer to. Obviously, closed source too, so can't look there. Basically not much help available. It's not like you can Google for an answer with an app that is this proprietary.

We later on figured out that the app requires the folder to be world writable so I wrote a separate script that I can push to change the NTFS permissions with CACLS.

Found a page with some undocumented features which allows to set group permissions (the documentation only mentions users)

http://support.microsoft.com/kb/162786

Being under pressure, my first attempt at writing a script to push a fix had a typo which caused bigger problems, but I quickly found and fixed it, and within an hour or so everything was golden.

The IT manager, who is a real asshole, barged in with his typical asshole look on his face and wanted an explanation for everything. That guy will ensure to take a bad day, and turn it into your worse nightmare.

Weekend yard work and stuff

2011-06-21T22:42:00.003-04:00

Was nice this weekend, got some stuff done, such as trimming my pine tree...

Also used a jackhammer for my first time... Did not get much time so decided to rent it for a week when I'm off, will be cheaper as this will probably take a couple days.

Oh Yeahhhhhhh