Log in

No account? Create an account
entries friends calendar profile eLarsson.net Previous Previous Next Next
home sick.. and bored... - Larsson Log — LiveJournal
grain of salt not included...
home sick.. and bored...
Who saw My Big Fat Obnoxious Boss on Fox this past Sunday? I'm not generally a fan of reality TV, but I can see myself watching this show religiously on Sunday nights. I realize that reality TV contestants are hand-picked for their cluelessness, but these people are supposed to be top financial and business professionals. It's really kind of frightening. "Mmmm.. This is the best champagne I've ever had" to a $5 bottle from the corner liquor store.. Priceless! Anyway.. if you didn't, try to find a repeat, and watch it.. even if you're not a reality TV fan... Arrested Development is off to a good start this season. Picked up the DVDs for season 1 last week, since I missed a couple of episodes. That's got to be one of the best written, hilarious, shows on TV today. Between these two, the Simpsons, and all the rest. Fox Sunday nights should be good this season...

This week.. hmmm.. Work Monday was uneventful. Yesterday, I spent a good portion of the day putting out fires, running between the two server rooms we have on campus with Paul trying to troubleshoot a series of seemingly unrelated problems. Now, everyone hates when this happens, but you usually learn something when it does... At the risk of oversimplification, what happened is that we thought we were having some communications problems between our core network switches in the morning, which resulted in us having to do a switch reboot at some point. What we didn't realize right away is that after this was done, some traffic (traffic on a few specific VLANs, to be precise) was no longer being passed between our two server rooms. This directly impacted our Novell iChain system, which I'm responsible for maintaining and basically front-ends and provides single-sign-on access to all of drewuniversity's web-based applications. This is one of those rare circumstances where our decision to improve service reliability by designing the system to be fault-tolerant ended up hurting more than helping...
For those of you not in IT, it might surprise you to hear that extra time and money spent on high-availability efforts can sometimes create worse failure conditions than not (now, i should add, that I don't regret our decision to do fault-tolerant iChain for one minute, it's saved our asses a few times already, but there are caveats...). Computers are really bad at detecting failure, or should I say partial failure specifically. That's what happened in this case. We had one iChain server in each of the two buildings. When connectivity was disrupted, the iChain boxes were no longer able to talk to all of their back-end servers. However, the load-balancing switch we have in front saw two healthy iChain boxes, as far as it was concerned, so it continued to distribute the load to both machines, instead of failing one of them out. IT people like to call these sorts of things "unanticipated failure modes," which is a really nice way of saying that "it f**ked up in a way we didn't plan for". The point is that you learn from it, and now I have a few ideas on how I could, with some extra work, tweak the load-balancer to, well, anticipate this particular failure mode.

To top this off, we also hardware failure on one of our Windows domain controllers that afternoon. The server decided it was a good idea to fry its network cards for some reason. (but this was s case where fault-tolerance paid off. The other DC just picked up the slack as it should.)

Today was much less eventful… meetings… Morning meeting on desktop videoconferencing... and an afternoon meeting with our webmaster on the University calendar..

Ended up going home at 2:30 because I wasn't feeling well.. Something at the University Club didn't agree with me, or more likely, it's just me... Oh well.. I think I'll make some tea or something...

Current Mood: okay okay

Leave a comment