General discussion

Locked

One of those days

By AmberHaze ·
Tags: Off Topic
blog root

This conversation is currently closed to new comments.

8 total posts (Page 1 of 1)  
| Thread display: Collapse - | Expand +

All Comments

Collapse -

Being shot in the foot.

by AmberHaze In reply to One of those days

<p>Ok, I may be known for little bit of arrogance in always knowing the answer for things, but the last 2 days, I just have to finally admit "Uncle"</p>
<p>First I guess I will summarize the actual incidents.</p>
<p>1. A new printer to be set up for one of my customers, and I overlook the fact that it is a GDI printer.  You guessed it, had to make it work with the old dos ACCPAC... enough said, wasn't going to work... but it took longer than it should have for me to realize it was a GDI printer.</p>
<p>2. Had to add another new workstation to a server for a client... Everything was working great, the users data was transfered from a stand alone to the network roaming profile... did the usual re-boot.... then boom... could not get the roaming profile to read.  Suddenly one of the accounting department's workstations started mis-behaving, and when I checked the network drive that contains all the accounting data... nada... then I checked the backup server... also nada.. all the backups had "disappeared"...  With my stomach in knots, I checked the servers directly to find the data was just fine.... at which point it dawned on me to check the number of CAL licenses, only to find that low and behold, we were 1 license short.  </p>
<p>3.  Had a phone system to re-program to handle a "cheap" fax in conjunction to a line with distinctive ringing.  Unfortunatly due to a unique combination of equipment, this simple task pruved to be down right ornery to get the timing right.  After a couple of hours of fighting with it, I discovered that the customer had a phone line which they didn't even know about (despite having been paying for it for a long time) switched the fax to this spare line... problem solved.</p>
<p>4.  Had to migrate a user to a new computer from an old one on a network served by a LINUX server.  Went to all the trouble of  using migration wizard, etc. only to discover that I had set up roaming profiles a while back for this customer, so did not need the migration file... Not counting the time waisted, the actual transfer took about 5 minutes.. give the new machine a machine account on the unix server, attach the machine to the network, re-boot, and then log in as the user... done.. as it should be.</p>
<p>There were a number of other incidents over the last two days, however I am sure you are starting to get the picture.  </p>
<p>Now that the week is over, I can look back and laugh about the whole incident, however in retrospect, I can clearly see the common theme running through all these incidences.  Lack of planning.  I was running in so many different directions that I was "jumping in" on situations without taking the time to properly plan these activities.  Overlooking the obvious, and letting my self get so wrapped up in the "bad day" I couldn't seem to turn it around.</p>
<p>In retrospect, an important lesson when things go wrong, take a step back, take a deep breath, and make sure you are planning your work rather than letting it drive you.. and above all... TGIF</p>

Collapse -

Being shot in the foot.

by Shellbot In reply to Being shot in the foot.

<p>I've had plenty of those kind of weeks! It can be so frustrating, especially when things that should be fairly straightforward end up being a big hassle to sort out.</p>
<p>If you've a good suggestion on planning, let me hear it :) I have my daily/weekly work planned out in advance, but every week, something throws a spanner in and my plans go to heck. Like you did, i end up running in different directions and at the end of the day i find i was not near as productive as i could have been.</p>
<p>Its good you can step back and laugh about it, sadly a lot of people can't.</p>
<p>Its Thursday, but I'm off tomorrow..so its my Friday (does a happy dance)</p>

Collapse -

A service call from the other place

by AmberHaze In reply to One of those days

<p>I thought it was about time I relate an interesting "trip" I had a few years back to get to get to a service call.</p>
<p>Specifically, we were installing a new copier / printer / fax into a school in the low arctic.</p>
<p>I guess I should start with some back ground information... Asside from working as a systems engineer, I used to work as a copier field engineer and on top of that, I also work am a weekend warrier in the SAR field (Search and Rescue)  At the time, I was a certified spotter and Navigater... (I am now a Search Co-ordinater awaiting a course to become Search Master.)  Due to these facts, I become first pick at our company on charters in the north..</p>
<p>Anyway, back to the story.  </p>
<p>7:00 am: The November morning in question started with my showing up at the airport on a nice brisk -40 morning (being in Canada, I specify all measuremnts in metric, but for those of you who are cenigrade impaired, it doesn't really make any diff at that temp... it is COLD!  All my gear (about 500 Kg worth) had been delivered to the hanger of the charter company the night before.  The Piper Senica we were chartering.  </p>
<p>It took about an hour to stow all our equipment in the stripped cabin of the plane. (Some dissassembly required to make things fit through the less than adaquite access hatch.  ) After proper checkout, we were ready to get under way by about 8:30.  Since I am a nav and regulations only call for 1 pilot under our circumstances, I was offered to fly right seat (which was a good thing since we had stripped all the back seats out...)</p>
<p>9:00 am we get clearance from tower and actuay get into the air.  Clear the zone and clime to our cruising altatude.  (At which point we note the outside temp is about -60.  </p>
<p>10:30 am  We are about half way between our destination which is Kinisao (Top end of Reindeer Lake) and our starting airport (CYPA or Prince Albert) Suddenly (in about 2 seconds) the cabin fills with thick black oily smoke.  So thick that neither the pilot nor myself can see the instruments... (At this point I am so thankful of my SAR training since my first thaught was procedure, rather than panic) We immediately closed the "heaters" which draw warmth from the engines and at the same time opened the "vents" While this cleared the smoke very fast, it also dropped the temperature from an uncomfortable 5 degrees down to -60 in the cabin REAL FAST.</p>
<p>We declaired an emergency over the radio, gave them our position and told them to stand by. We started investigating to discover our oil pressure on the right engine was faltering... Feathered back the engine to see the pressure drop off completely... performed a shutdown... looked like we had blown a seal of some kind... have to wait to find out later that we had lost the front prop seal.  With just one engine, we were now in an emergency situation... Needed to land ASAP since we now had no margin for safety. (To put it in lay terms, we were comming down one way or the other so better to put down by choice than the other way)</p>
<p>This is the point where I reach the amusing realization that I am point of contact for the local SAR, so my home would be getting a call to go to stand by.</p>
<p>We notified Nav Canada of our situation and instructed we wanted to attempt to make La Ronge (although up-wind, at least they have something of a town and maintenance facilities.)  The Trafic Controller informed us that due to weather and a few other reasons, they were "Strongly recommending" a remote strip in Sandy bay instead.  The local authorities were allerted as we attempted to bring the plane around to the strip in question (20 minutes flying time on the 1 engine) Fortunately, we made it to the strip and made the difficult landing. (with no chance of pulling out and trying again if we wanted to...)  The RCMP had an officer on scene awaiting our hoped for arrival, and after secruing the plane, he took us back to thier offices.  </p>
<p>11:00 We arrive at the RCMP office for Sandy Bay and make all the appropriate phone calls.  I let my office know what is going on, and the Pilot calls his company and tells them to send a plane to "rescue" us.  For those of you (probably almost all of you) who are not familiar with Sandy Bay, Saskatchewan, it is a small town way beyond nowhere.</p>
<p>3:30 pm our plane from the charter company finally arrives and we are taken back to the little strip.  It only takes us about 1/2 an hour to re-pack the new plane. They had send a Piper Chieftan with 2 pilots and thier chief mechanic.  While we loaded the new plane, the mechanic looked at the damaged one and informed us that we had infact blown the front prop shaft seal, which was pouring oil onto the exhaust manifold... The increadable cold was the only thing which had prevented it from "catching fire" while we were in the air..... </p>
<p>You would think this is the end of the "adventure"... fooled you...</p>
<p>4:15 pm we are back in the air and continue on our way to Kinisao.  We quickly discover that the cabin heaters have tripped a breaker due to the outside temp, so we basically have no heat in the cabin... So I burry myself in a huge pile of blanckets and "snuggle in" for the cold ride north.  I should mention that the battery on the plane was dead, and so the RCMP cruiser had to "boost" the plane to get it started.</p>
<p>5:00 pm we arrive at Kinisao (Thanks to the size airplane, our travel was much faster than before) The truck is awaiting to haul all the equipment and myself to the village it is going to (about a 1 hour drive). I ride in the back assembling the equipment so that when we arrive, we can drop it off ready to install and test.  Meanwhile the air crew get the plane prepped for the return flight, and have some supper. (Note I haven't mentioned eating yet?)</p>
<p>6:30 pm We arrive at the village, and I do the fasted install I have ever done in my life, including adding 1 network run, adding 1 phone line, adding printer drivers to 7 computers, and testing... all in under an hour.</p>
<p>9:00 pm we  get back to the Kinisao airport to find the plane has been let to go "cold" as the airport manager protested thier keeping an engine "warm"  You should understand that this airport is actually a little strip with a small cabin for a terminal and no on site staff past about 5 pm.  We load the "returning equipment in to the plane, and all pile in.. to find the battery is dead again. (go figure)  Fortunately the truck has not left yet, so we get the driver to boost us getting the left engine going... all well and good, so he takes off for home.  I am sitting beside the mechanic while the pilots are running through starting the second engine when I hear him swear and yell NO! as they proceeded to make a mistake in proceedure and "flood" the right engine... making starting it next to impossible.  After trying to start it for about half an hour, the crew moves back to the terminal building and the mechanic proceeds to build a "tent" over the engine and scrounges a tundra tourch (picture a tiger torch on steroids) from I don't want to know where... about half an hour later he and the pilot come running into the building, grab every fire extinguisher they can find fast, and run back out... not a word....  I think the theory was to warm up the engine to both evaporate the excess fuel, and at the same time make it easier to start.</p>
<p>2:00 am  After several false attempts to start it, the mechanic declairs it's now or never... We all pile back into the plane and the mechanic does the startup... and it catches... we now have 2 working engines agian (This whole time, the right engine is idling to keep warm, and to generate power.)  We quickly get in the air before anything else can go wrong... oops... remember those heaters... sure enough, 5 minutes in the air.. and they die again.  </p>
<p>4:00 am With no more incidents, we got back to PA safely.  When I got home, I made a quick call to the office and left a message not to expect me the next morning.</p>
<p>One last note, in case you are wondering about the title of this blog, I would have called it "A service call from H-E- dourble toothpicks" but given the -60 temps we were dealing with, it would have implied that it had truely frozen over... </p>
<p> </p>

Collapse -

Don't forget to think.

by AmberHaze In reply to One of those days

<p>In my current role, I support mostly client networks with about 80% of the work done remotely.</p>
<p>Well over the last 2 days, I had a humbling experience which lead me to realize that it is very important to practice what you preach.</p>
<p>I guess I should start by giving you a mental image of our own offices network topology.  We have 2 servers, 1 (Brand new) is 2003 which is providing AD, MicroSoft SQL, and IIS.  Our other server is an older Linux Slackware box which provides everything else, (Apache, Sendmail, FTP, SSH, Remote Backup, and Remote Network Monitoring) We have a total of 4 subnets within the company with a collection of "cheap" routers and switches holding the whole mess together. Admin, Backbone, Services, and Isolation (for potentially infected or otherwise compromised customer machines)</p>
<p>Up until recently, this all operated quite happily mounted upon a single Static IP address, however due to the addition of some unconfigurable windows applications which need web presence via IIS and the balance of all our web presence being on our UNIX box, we decided it was time to finally add the second Static IP and start dividing out some services..  </p>
<p>Two weeks ago, our ISP supplied the new static, appropriate services were re-configured, and all worked as normal.</p>
<p>Now comes the fun part. On Wednesday morning, we noticed the isolation network was very slow and our main admin subnet was not a lot better.  This was immediately attributed to a badly infected computer which had been pulling all the bandwidth it could from a normally "safe" port.  We took the normal protective action and thought no more of the incident.</p>
<p>A couple of hours later,  we started loosing services in what seemed like  a random fassion.  The diagnostics logs were not making much sence and I suspected that with the IP split of services that I had overlooked some kind of conflict.  Normal outbound clients were working just fine, but it seemed that all our external presences were dead in the water.  </p>
<p>I started line by lining the configs of each problem service looking for a possible conflict.  Tweeking things here and there that I thought might have been part of the problem.  After about 2 hours of this, I was in worse shape than when I started.. now all services were dead to the outside world although they all were still working perfectly inside the network.</p>
<p>I then began suspecting the backbone switch and routers could be the source of my troubles... Fortunately we are a supply company as well, so I went into the supplies wharehouse and took an armload of gear and literally replaced the entire backbone.. Complete with related router re-programming, etc.  (I should add that at this point, panic was starting to set in since I look after some 50+ remote networks, and with all my automated services being down, I was both blind the the whole status as well as I did not have many of my custom tools available to me as they largely rely on our services being funtional.)</p>
<p>As you probably already guessed, the above did not solve the issue.  </p>
<p>One of my co-workers (The sales manager) pulled my from the racks and made me sit down and have a drink and "talk through" what was wrong. he knew I was stressed and not making any headway on the problem.  It didn't help knowing the phones were ringing off the hook by cusomers wanting to know what the warnings they were recieving were all about.  (We have an automated monitoring system with all our client networks, when they loose contact with our servers for an extended period of time, they automatically email local contact people to allert them that something is wrong... Kind of an automated trouble system.)  This break gave me the perspective I needed.</p>
<p>Now the lightbulb finally went off... what if the problem was not with our network after all... what if it was at our ISP?  A quick call to thier NOC and some quick tracing later and sure enough, we know the problem is thier end, not mine, but they are not sure what it is yet either, but we did confirm it was in one of thier routers.  It is now after 5pm and there is nothing more I can do, so I go home and have a miserable night as I know all my remote backups will be failing and I will have literally hundreds of trouble tickets in the morning to follow up on... fortunately most if not all will simply be a matter of a quick phone call to reassure clients the problems were due to our end, not thiers.</p>
<p>8:am the next morning... I am immediately on the phone to the NOC to find out  what they found out... (Still down by the way...) They assured me they had the problem licked, and that I should expect normal services to resume in about 1/2 an hour.. they would call as soon as they had finished... and explain everything.</p>
<p>The longest half hour I have had in a long time occurs.... I take advantage of it to restore by backups of all my router and service configs, just in case any of my "tweeking" broke anything.  (At least my auto backup of EVERYTHING, including router configs, worked perfectly.) Then my call came...</p>
<p>I guess two weeks earlier when we added the second static IP to our service, the sales department messed up the paperwork somewhere.  While the work order to install and set up was completed properly, thier accounting system did not get the info.  Thier accounting system runs a service audit periodically, and when it found the discrepancy, instead of doing the logical thing of allerting someone to look into it, they have it programmed to **** off all settings it can't expalin.  As a result, the audit server blew off some of the routing tables which pointed to our statics.  While the static ip's were legit, it was hit and miss wether you could reach them externally. This meant we could reach outside and the tcp/ip pipe initially tested ok (which is why I thought internal problem in the first place) but that progressively as data rippled out from thier audit computer, we progressively experienced more and more inbound failure.</p>
<p>The moral of this experience?</p>
<p>1. Backup backup backup (whew) (While my system worked perfectly</p>
<p>2. Don't forget to look after your own equipment (More of an issue for outsource providers such as myself)</p>
<p>3. If all testing says it isn't your fault, don't be afraid to believe it and look elsewhere.</p>
<p>4. If panic sets in, take a step back.. go have a coffee (or something stronger) and carefully examine what you know.  Then make a systematic logical plan to proceed before you do anything more.</p>
<p> </p>

Collapse -

Cold enough to freeze the brass balls off a Monkey

by AmberHaze In reply to One of those days

<p>About a month back, I had a network installation which on the surface of it looked pretty cushey.  2 days to install 21 runs in a remote location. (A summer fire cache) No actual computer work. Just to install and certify each run.  I had a fishing cabin booked, and was looking forward to the change in pace for a couple of days.</p>
<p>On the appointed day to drive up to the site, it turned to about -50 before windchill.  (I am Canadian, so I am talking Celcius, however at that temp, it really doesn't matter....)  I figured go and get the job done anyway.</p>
<p>Arriving at my cabin late at night, I discover that it is about the size of a bus stop, and about as good at keeping the cold out.  I burned about 2 cord of wood that night, and my water on the side table still froze by morning. Oh and did I mention that there was no washroom facilities of any kind?</p>
<p>The next morning, I continued on to the job site which was about another hour north.  Uppon arrival, I discovered that the contracter who had removed the snow from around the ATCO trailers had only done the one end, so I would have to crawl 40' under the trailer for some wiring... Grrrr.</p>
<p>My next chilling discovery was that even with the furnaces working full tilt, these summer trailers were keeping about +5 degrees celcius (that would be about 38 or so for you folks down south.)  I was not a happy camper.</p>
<p>The next "fun" discovery was that the wiring cabinet where I was to install some wonderful brand new wiring racks was 1/2 an inch too narrow.</p>
<p>After a couple of .... phone calls (was going to say quick.. but that would have been an exageration) I recieved approval to modify the new wire racks to make them fit.  So driving about an hour down the highway to a highway's maintenance facility, I found a welder and did some sweet talking to get some quicky, on the spot welding done. (It helped that the work I was doing was also for the governernment.)</p>
<p>I get back to the site and install the now modified racks... now a perfect fit... </p>
<p>I get the first couple of easy wire runds installed and realize that it is getting dark, and look around for the light switch, to discover that while I do have power, I don't seem to have much in the way of working light bulbs.  Time to quit for the night.. and head back to my frozen cabin.  While more comfortable that before, i am definatly not a happy camper, and it was a good thing I was working solo on this project as without washroom facilities, I was in bad need of a shower.</p>
<p>Next morning, I aim at being back on the site at sunrise to give myself as much daylight as possible, since I am already a half a day behind on the project, and am suspecting there will be several other slowdowns.</p>
<p>I get all the exterior runs to the 2nd trailer done only to find that due to the intense cold, when I was stapling the lines in place, the staples would shatter the cables... rendering about 300 feet and about 4 runs to be complete garbage.  I quickly re-strung all the runs, and decided to leave the hanging to the next day, hoping I would have a briliant idea that night which would allow my to stay in code, and restrain the wires, without having them crack on me.</p>
<p>Due to my terrible state in not having had a shower in 3 working days now, I decided to examing the map and discoverd a small town in the next province... only an extra half hour drive in the opposite direction to where I had been staying...  All I can say, they had a real hotel, and I spent about an hour in the shower, then another in thier restraunt, enjoying this vestige of civilization.</p>
<p>I am now a full day behind schedule and again arrive back at the site as early as possible.  Over night I had a brain storm regarding using wire ties and a system of suspension hangers so that the wire runs would be seccured withoutthe destructive use of staples... which was a good thing since it was so cold, in a test, warm wire was freezing to the point of shatter with just about 30 seconds of exposure.</p>
<p>After spending half the day installing my improvised hangers, then terminating and testing all the runs, I was needless to say, exhausted.  But still had 3 runs to install at another building in the area... the local conservation officer's building.</p>
<p>Fortunately, this was a standard modern building with enclosed crawl space that you could actually stand up in, underneath.  3 hours later I was done.... except in loading up the truck, I accidentally let the door close and lock.... with my coat inside... (The local staff were doing a presentation at the local high school) While my car was not warm, it was better than standing outside in -50 weather, so I drove to the school and had one of the officers lend me his key so I could get my coat.  </p>
<p>One day late, and physically much the worse for wear, but the job as done....  And to think, I originally figured this would be a cushey one.</p>
<p>.</p>

Collapse -

Cold enough to freeze the brass balls off a Monkey

by Shellbot In reply to Cold enough to freeze the ...

<p>I really hope you love your job! I wouldn't be able to take it. Where on earth do they keep sending you? La Ronge?</p>
<p>I'm sorry to admit, i laughed..i seen -50 and had a good howl, but i know it not that funny (having grew up with those temps). </p>
<p>Fishing shack :) Next time take the hotel right away !</p>

Collapse -

Cold enough to freeze the brass balls off a Monkey

by AmberHaze In reply to Cold enough to freeze the ...

<p>That particular one was between Pelican Narrows and Sandy Bay... La Ronge wouldn't have been so bad.</p>

Collapse -

Back from Vacation

by AmberHaze In reply to One of those days

<p>Well, I just got back from vacation... so I guess it is time I write something since I have not written in a bit.</p>
<p>Today's story... KISS.</p>
<p>I was doing a connectivity on a till yesterday... (Yes just a simple boring Null cable on an RS232 port)</p>
<p>To make a long story short... brand new till, brand new cable, old computer.  So what did our boy genius start with on the diagnostics... confirmed the settings on the till... confirmed the cable... Checked the coumputer... yep the old computer was crap... after waisting about 2 hours on the above, I realized I was waisting my time, grabbed a different computer from the client's office, and it worked instantly... told customer the old computer would need to be repaird or replaced (our company was responsible for the till system... not the computer.)  </p>
<p>What equipment was this computer?  An old Packard Bell with a 200Mhz cpu and 98 release B</p>
<p>Given the KISS approach... should have simply tried a different computer right away given all the other equipment was new (and worked in our office at setup time.)</p>
<p>I find myself sometimes trying to be "nice" in the field when people have older equipment, in trying to repair something that is woefully inadiquit.  In reality what I am doing is prolonging the agony of the staff in having to deal with equipment that is inadaquit.  I guess the lesson to remember is focus on the technical solution instead of trying to be "nice" and give the bad news as gently as you can.</p>

Back to After Hours Forum
8 total posts (Page 1 of 1)  

Related Discussions

Related Forums