General discussion

Locked

50m3 r@nd0M +h0ugh+s fr0m Joe

By Joseph Moore ·
Tags: Off Topic
blog root

This conversation is currently closed to new comments.

17 total posts (Page 2 of 2)   Prev   01 | 02
| Thread display: Collapse - | Expand +

All Comments

Collapse -

Didn't get the "today is DNS problem day" memo

by Joseph Moore In reply to 50m3 r@nd0M +h0ugh+s fr0 ...

It all started Saturday night. My production rollout of Release 2.1 of a public website earlier that day went just fine. I was able to leave work at 1PM, go home, meet my girlfriend for an early dinner, then off to Revenge of the Sith.
We got home at 10PM, and that's when the DNS problems started. While I was at the movie, the Operations desk called me (like a kind movie viewer, I turned the tones of my phone off) and said that all of the company websites I monitor were coming up Red in the monitoring software.
Can I check it out?
Sure. I'll try.
So, visions of jedi roasting by a volcano running in my mind (or more like crawling across charcoal with one arm!), I fired up my laptop and connected over the VPN.
Sure enough, the monitoring was all ablaze with red alarms. Everything that had a www or ended in .com was coming up as being offline.
Turns out the fun was due to a misconfigured proxy server deployed by our sister company. Their proxy was using our internal DNS servers for resolution, and their proxy was doing a DNS story, trying to resolve something I'm still investigating (looking up a PTR record from a DNS server that rejects PTR lookups! How rude!!!).
This also caused, into Sunday, e-mail delivery problems, since the SMTP servers couldn't resolve the MX records for the other internal SMTP servers and Spam servers.
So, a proxy server brought down e-mail, internal monitoring, and freaked out Operations, and ruined my RotS buzz!
sigh....

Collapse -

Making a server while eating Rice Krispie Treats

by Joseph Moore In reply to 50m3 r@nd0M +h0ugh+s fr0 ...

Right now, as I finish my last Rice Krispie Treat (yum!), I'm also building a new Win2K3 Standard server. New server for new price feed data stream application. What fun. Done this so many times over the years....
It's an IBM xSeries 336 server, so I'm using the IBM ServerRAID cd and the other IBM cd that does the actual work. I just had to check a few checkboxes, put the Win2K3 cd in for 12 minutes for the i386 folder to be copied to some temp space, and click NEXT a lot.
Gotta love how people still think that server builds are laborous.
It's our little secret. Shhhh!!!!
Think I need a soda to wash down the marshmellow.

Collapse -

Making a server while eating Rice Krispie Treats

by Jay Garmon Contributor In reply to Making a server while eat ...

Mmmmmmm...rice krispie treats.

Collapse -

Making a server while eating Rice Krispie Treats

by jmgarvin In reply to Making a server while eat ...

They are laborus!  Clicking that Next button is exhausting work!

Collapse -

Playing "find the server" game today. Weeee!!!

by Joseph Moore In reply to 50m3 r@nd0M +h0ugh+s fr0 ...

Ok, it's one of those crazy days where you get assigned impossible
tasks to perform, and everyone is waiting on your results in order to
finish some even larger and crazier and more impossible project.<br />
Yes, in other words, the typical day.<br />
Today I was given a hide and seek quest by the accounting department.
They wanted me to find a server with a specific asset tag # on it. The
description with the tag # was the name of my NT4 PDC (yes, we still
have a couple NT4 flat domains!). I thought this would be easy. Just
verify the tag #s match, and then tell accounting that yes, we still
use that server.<br />
They wanted to see if they could take it off the books, like we do when older servers are decommissioned.<br />
Turns out it wasn't that easy. The asset tag # didn't match. Plus I
figured out that the tag # they want me to find was issued around March
1999.<br />
I started working for this company in 2001.<br />
I checked  the server room and pretty well concluded that whatever
machine with the quested for asset tag # must be a really old one,
since other servers with slightly more recent tag #s are all either
phased out or being phased out. You know, those 500MHZ PIII servers.
They're history around here, what with the 2.8Ghz Xeon Blade servers we
typically pick up.<br />
So, I still have a missing tag #. I checked the rest of the servers and
a good portion of other devices in our large server room (that was a
couple of hours work right there) and didn't get a match.<br />
So I get to go to our offsite DR facility tomorrow and see if the asset
tag is on something there. Then once I (maybe) find it, accounting can
change the description in their precious spreadsheet to match that
intangible concept known as Reality.<br />
sigh....<br />
At least I found a bag of Cracker Jack. Yum! Candy! I love candy.<br />
<br />

Collapse -

Oops!!!!

by Joseph Moore In reply to 50m3 r@nd0M +h0ugh+s fr0 ...

Yes, the dreaded phrase in the IT world. It happened here at work today.<br />
A co-worker (it wasn't me this time! Promise!!!) connected a new server
into one of our older server cabinets today, and plugged it into an
overworked power strip.<br />
He then turned the server on.<br />
And there was this <em>POP</em> sound, then silence.<br />
<br />
The power strip was rated for 12 amps (I think it's amps, or is it
volts; I always get that confused), and it apparently was already up at
about that number. Bringing the new server online pushed the strip
passed 12. Which tripped the breaker in the strip, killing power to
it....<br />
<br />
And to the various servers plugged into that strip also!<br />
<br />
Hence, the Oops!<br />
<br />
Some of the servers were dual-powered, properly plugged into other
strips, so they were OK. But some of the servers were older ones with a
single power supply. And it's those ones that brought up the alarm. My
monitoring software suddenly started flaring red, with various servers
being offline. They were older servers, but my machines just don't shut
themselves off like that, unless something bad has happened.<br />
<br />
So I went into the server room, found the servers that were registering
as being unresponsive, and saw the lack of power truth. My co-worker
was still in there; he was doing something else and hadn't noticed the
outage. I went over, unplugged his new server (I knew he was mounting
the server in the cabinet today, so I pretty much had an idea what
happened when it happened), and flipped the breaker back on. <br />
<br />
Voila! Power restored. <br />
<br />
I then took a few minutes, and made sure everything that went down came
back up again. Monitoring software went calm green again, and we're
waiting around to pick up any of the pieces. So far, the only fallout
was our Linux anti-spam servers went down, and one of them didn't come
up properly. SMTP on it was working, but the actual Spam filtering
process didn't. But that's OK now.<br />
<br />
Welcome to Monday in the trenches.<br />
<br />
Pop!!!!<br />

Collapse -

The Sky Is Falling Friday....

by Joseph Moore In reply to 50m3 r@nd0M +h0ugh+s fr0 ...

Boy, today's been fun, and it's only lunchtime!<br />
Hardware failure on an application server (drive in a RAID1 failed) made a website unstable....<br />
A rollout of a new web server (that we finished at 10PM last night) for another website had to be rolled back.....<br />
Another application (that I don't deal with, luckily) had bandwidth
issues all last night, so those who deal with it are all grumpy
today....<br />
Remote admins of a FTP site locked themselves out and had to wait until I came into the office to re-enable them.....<br />
And people are getting a QA environment confused with a Dev environment, when the two are mutually exclusive.<br />
<br />
I'm just glad I went and got a couple cheeseburgers. Plus some good Swiss dark chocolate.<br />
<br />

Back to After Hours Forum
17 total posts (Page 2 of 2)   Prev   01 | 02

Related Discussions

Related Forums