Long day

I try to avoid working on most sundays. but it can be pretty dang hard when that's the official maintenance window. Even so, it's not usually that big a deal. But today we needed to add some tier 3 storage to our SAN. Man is that stuff expensive. Our tier 1 probably costs about $30k per TB. I could be high there, but we paid that much for 10 GB of tier 2 which is crappy SATA vs. fiber drives on our tier1. So, since we need space but cash is king, queen, jack, ace, 10, 9, 8, 7, 5 (but not 6), 4 & 2 (nope. 3 neither) we added our own disk array to our virtual storage software. Slower and more complicated, but much much cheaper. Only $10k for 12TB. We could've possibly done it for about 4k if we wanted to build it ourselves. But, by the time you factor in things like carrying spare parts and downtime it starts to not look like a good idea. We actually tried a Drobo Pro quasi-SAN, but it couldn't produce IDE speeds. Our needs were modest but it failed abysmally in that area. And we had that thing loaded too. It had 8 2TB drives for a total of 16TB raw. pretty impressive for a little unit. We got all that for $3500. Unfortunately, it didn't do that job so that was that. Working on enterprise systems is so weird because the premiums for this stuff is insane. But the industry works on supply and demand so the price is high. Yeah, i know this is boring. It wouldn't have taken so long but our system was configured oddly (at best) and so when we hooked the array to it, Windows wouldn't see it. After much trial and error my cohorts in crime figured out that the setting was configured incorrectly (a setting that they inherited when they took over the system). Once they figured that out, we were golden. Well, brass at least. I wasn't actively involved (thankfully) with that ordeal, but I had to shut down the virtual environment. 60+ servers on 6 hosts don't go down (or up) quickly. But, thanks to the beauty of technology (vmware), about all I had to do was Ctrl-A (select all) and right click and tell them to shut down and then watch watch watch them. Then we moved some stuff around and when we were about all done we powered up one of our beefiest servers (32 GB of RAM w/ 4 dual core procs) and it blue screened in a way I had never seen before. Unfortunately, this server was purely physical so no leveraging the server onto other hardware without pain pain pain. And this was the second most important server in our enterprise. We have one whole division that screams bloody murder if they can't get their reports every few hours. A day or two outage would be brutal. So, we are all looking at each other thinking expletives in our mind. And we're all of one mind wishing that this had happened a year later. But, time stops for no company and the economy be damned the server was almost four years old. Of course, having a piece of hardware that costs $20k minimum to replace is hard to justify when blood is being spilled in the trenches. So, the only clue we have is a red light on the front of the server. After doing some troubleshooting, we figure out that it might be the RAM that's bad. Well, to be honest, I pulled the server out a little and it had pretty yellow lights on DIMMs 5 & 6. It was kind of cool on the inside. Every proc had its own cell board complete w/ RAM and more pretty yellow lights signifying bad things. Unfortunately, swapping the RAM around only proved nothing. Oddly enough, yanking the RAM for the entire board allowed the server to boot. So, the server is up and running with a lowly 24GB of RAM. oh well, HP's gonna love us demanding the send us a new board. We paid about $800 for the warranty. Good investment me thinks.

ok, if you actually read this post pat yourself on the back. If you read it without falling asleep (nodding off is ok), then you may have a piece of chocolate. If you read it, stayed awake, and understood what I was talking about then you might consider getting a life. I hear e-bay is a good place to start.