IT Employment

Ask something
Discussion Locked

Hardware in a Maintenance Room

0 helpful
11 replies

by ippirate 2 views

Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Here's my situation. I have a switch stack and a router/firewall that are being stored in a maintenance/janitorial room. This hardware sits in an open frame wall rack, 4 feet from two industrial air handlers. There is a myriad of chemicals in the room as well giving off vapors, a mop sink and two large industrial power panels.

All of my backbone hardware as well as my phone equipment sit in this room measuring 8x15. Here's a surprise, there are always issues with this equipment. The router resetting, DHCP service being lost, no pass through of SMTP/POP3 traffic to the outside world and the list goes on. I am freaking out, have been since I got here about the location of this equipment. Management keeps saying, just fix it and find out what is causing it. Me: The environment has a majority to do with it. Them: No, it doesn't

Here's what I am asking.

Anyone that is willing to comment on this and confirm that the environment can indeed cause these issues on the level of hardware problems, please email a response to the address listed below. I can't effectively troubleshoot the problems because the fixes are so random, this leads me back to something that I either am not or cannot address/mitigate. The few pieces of new equipment that have been installed have been fine for a few months and then the problems begin to occur, increasing over time.

Please, help me get through to these guys.

webmaster@transnav.com

11 replies

by jdmercha

The problems you describe would lead me to check out two things.

First. How's the temperature in the room? Ideally 68 would be nice but real problems shouldn't occur until you hit 80. (though equipment lifespan can be shortened by higher temperatures.)

How's your power supply. Most power companies do not supply the steady voltage needed by swithing equipment. You should have everything in that closet powerd through a line stabilizing (I forget the real techno term) UPS.

by bubbaonthenet

It's called a line conditioner. APC makes one Line-R and they make a wide variety of models. The SmartUPS model of UPSs have Line Conditioning built in them.

by bubbaonthenet

DMarks are usually LOUD (Water Conditioners, Air Compressors, Electric Mains, Fuse Boxes, that sort of thing), nasty, spider filled rooms that no one ever visits. I've seen some nasty DMarks. I?ve seen them with Emergency Venting that essentially opens almost one exposed wall to the outside environment.

In fact, I have never seen one that has been treated by HVAC even while working for a Hotel Casino. It makes no sense to make the room comfortable for humans when it will not be inhabited by humans. Copper wire punched down to a block does not need the same requirements as a Server does. It just needs to be dry and re-punched down correctly once in a while as it loosens or becomes shorted out by other leads. Just as long as it does not get over the same temperature as the CPU does inside of a Computer (120-140F) you should be fine.

I would not recommend running a humidifier in a room with the main power lines to the building --if you can't see the reason then God Bless.

The rule of thumb is that if the Telephone Company is OK with the DMark then whatever else in there should be ok as well. They have SERIOUS money invested in their hardware as well so if an environment is unfit they will not install.

For the most part how many times has the DMark in your OWN HOME faulted your lines, disconnected a conversation or any other weird anomalies? When DNS does not solve a name or DHCP is not handing out licenses we do not send out Hardware or Helpdesk Technicians to start tracing Copper, we look at the Server to see if DHCP has licenses or the services are running et cetera.

It?s usually when it gets wet or overheats due to extreme temperatures that a DMark can be to blame --which it will be VERY OBVIOUS as the equipment will be wet or it will be too hot to be in there.

I would suggest the following:
1. Check your DMark for any obvious abuse or obvious problems (dripping water or water damage on your punch down boxes.)
2. Any exposed copper on the wiring --making short circuts.
3. Is the wiring leading to/from the building lying across anything that may interfere with Data Communication: the little transformers in Florescent lighting wreck havoc, look also for CAT 5 across power leads.
4. Are the pulls from the room to your Server Room in healthy shape (see if your company will rent a Fluke tester to check those lines [to see if the pulls are too long, the connectors are wired right, et al.]
5. Can you move most of this equipment to your Server Room and do the data pulls to the offices/cubes from there? That makes the most sense as you do not have to run from room to room when you are troubleshooting Network Issues.
6. Bring in 3 Network Service Companies (like BlackBox or any local Joe) to give their professional opinion on how your Building is wired. Ask them if it is in and up to your Local Codes --this is free to you because they want to rewire your building (see if their diagnoses are uniform.)
7. Purchase tools to troubleshoot the real issue at hand: If you are dropping packets or information is not getting in/out purchase or rent a Network Sniffer to give you the real scoop on what is going on; buy the right tool for the problem at hand. Sort of like if you keep burning things in the oven purchase a Thermometer --USE COMMON SENSE.

No DHCP [if it is on a Server, check the services, check the event log (NT I am assuming) if it is controlled by a home router that runs a DHCP Service are you running out of IPs? Do the simple troubleshooting steps as directed by the manufacturer.]

--Your job is to find out the real issue and inform your Management what you think it needs, what it does and what will be affected, and the costs associated.

8. Contact your ISP if you feel certain protocols are being blocked ?you have a firewall, did you accidentally block SMTP one afternoon?
9. Relax, step away from the DMark and take one problem at a time.
[Mike Dupris]

by ippirate

bubba,

That is the problem. I have been taking each problem one at a time. There doesn't seem to be any real ryhme or reason. No consistency in time/occurence. No consistency in problems, sources or solutions. Things tend to be quite random. I would lean toward at least some degree of consistency, the only thing that there is there is the fact that things keep queering out, some small, some large and they have gotten progressively worse over time.

Thanks,

by bubbaonthenet

Here is how I approach it at small shops where you have to be a jack of all trades:
1. Take each issue and document the exact problem, work on resolving it and document the fix [build a knowledge base.]

Building a knowledge base is so beneficial on saving time and justifying purchases as well as hiring additional resources (it also identifies where you spend the majority of your time.)

2. When you have to fly solo you are sometimes too involved with the current problem you sometimes forget to step back to see if there are other forces influencing the issue at hand. So step back and refer to other issues and give them some investigation and absolve them.

Remember you can get fried out too easily as being the only guy or 1 of 3 or 1 of 2. Take vacation time. Take time out for you. Do whatever, but your mental health is just as important as your skill sets when troubleshooting.

3. Your in charge of a lot more then you think, you have to remember that there are hundreds of responsibilities and thousands of tasks with running an IT Infrastructure and maintaining their systems. You do not work in a Large Corporate structure where you have a given scope of responsibilities and duties, where you report at 7, finish your day's routines by 11:14AM and spend the rest of the day meeting and philosophizing about what "Ideal IT" is.

Sometimes things are not going to be the result of another problem's having cascading effect --if at all. You may have your Exchange Server go down one minute and then an applet at a website your sales people always go won't work. Keep perspective: Start out with basic troubleshooting rather then looking at the least likely like the DMark.

4. Remember that this job really is more then just working with Computers, it is how you can balance in an ever changing environment (most of it in the control of your users hands, a tiny portion in your control and the remainder of control in the hands of outside entities --remember that when a virus pays you a visit or an upgrade turns in to a nightmare.)

Maintaining your cool and confidence in what got you to where you are today; being honest with your capabilities (remember IT is NOT about Ego it is about Computer Technology --NO ONE in this industry knows everything, NO ONE does, and those who claim they do are loose canons.)

Never be afraid to say to your Boss I do not know but I will find out --he hired you knowing your limitations and he expects you to grow, so take this chance and run.

5. Remember, the less resources or people there are to support a Network the more things will go wrong as the systems become neglected. Keep the Manpower Ratios (Server to Manpower, Users to Manpower, Maintenance to Manpower, and Workstations to Manpower) always in Mind. You always should be proud of what you accomplish even when you feel you are not doing much.

OK? That's the pep talk. Now lets trouble shoot your problems. Tell me about the DMark again:

A. What kind of hardware do you have? Give as much facts as you can.
B. What kind of DHCP is running (appliance or Software?)
C. Post good snapshots of the DMark on a page on a website and perhaps someone will be able to instantly see an issue.
D. What sort of maintenance have you done? If the hardware has fans are they clean and running --that sort of thing.
E. Have you tried configuring and swapping in/out new or similar hardware?
F. Do not be afraid to list your level of expertise with the hardware as someone may give you "advice" such as a good book or a beneficial course to help get you to where you need to be supporting your environment.

Consider joining a local HelpDesk Chapter -although they are geared to Helpdesks it is beneficial for Administrators like us to meet and Network with others [http://www.thinkhdi.com/]

List all of the variables and give your picture color with the make and model information because there are geeks out there that could tell you similar woes with a product or what they did to get things running.

by stress junkie

The equipment that you mention will come with a description of the environment that you need to provide to see the equipment work properly. The janitor's closet with a sink and a mop and bucket will most likely be humid. I'm sure that all of your equipment will provide both a maximum acceptable level of humidity for its environment and the documentation will include the requirement that water never condenses in the equipment environment. You may have water condensing inside the equipment cases and getting on curcuits or connectors.

by stuck_in

Do one type of problem occur at a set time or day or after another action? Even the madman has his logic. Check the environmentals! Electric, humidity, etc. How close are the electric panels and does any equipment create a huge draw of electricity throughout the day. I once had a power spike/brownout problem where the transformer down the street was the cause. Only by using the PowerChute software from APC was I able to prove to the electricity company that they were the problem. A couple weeks later they switched out the transformer and my problems disappeared.

After the mopping is done at night; does your equipment flake out a few hours later? (hih humidity incidents) You mentioned that even new equipment slowly gets worse over time. How long are we talking? Weeks, months? When was the last time you blew out the dust in the equipment?

by daniel.muzrall

I'd suggest 3 things:
1 - Determine your power needs, and install an appropriate number of properly sized UPSs with line filtering. Sensitive equipment like switches and routers like constant, stable power. A UPS, such as the APC SmartUPS product line, will provide you with a constant 120v, and power output is forced into sinesoidal wave form at 60Hz. You may want to consider getting an enclosed rack with a SmartUPS and power distribution units built in.

2 - Check the humidity. It may sound like a simple thing, but if it's humid, install a dehumidifier. Spend a couple of extra bucks to get one of the self emptying/draining ones, particularly where you are in a janitorial closet which has a sink and drain.

3 - Evaluate the air handlers. Are the units simply in the same room, but not providing ventilation to that room? If not, contact an HVAC vendor to tie that room into the HVAC system. If you do in fact have high humidity, high temperatures, and volatilized chemicals wafting through the room, an HVAC system (or even an exhaust fan to the outside) will do you a world of wonders.

Good Luck!

-Dan

by rajwar2000

The operating environment of the equipments has to be maintained properly. proper cooling and ventilation is required for the HW to remain stable.
First define the problem statement,update the firmware on all devices, check for operating environment factors,compatibility issues and then sort it out.
Normally i have seen devices behaving erratic due to improper cooling, they hang etc.

by cabound

I read your post and all of the replies. Here is something that I had happen at one of my customers....
I get a call from another vendor. They have been having very similar issues to yours on a network that they pulled in. They had been out to this customer several times, cannot seem to find the problem. I drop in there and they have placed the router/firewall device on top of a cabinet, within 4 inches of a fluorescent tube. These things generate A LOT of noise. That will cause all kinds of erratic and untraceable issues within your network. I moved the device to a shelf on the wall and their problems went away. Check and see that your data electronic equipment is no where near to any large electrical equipment that might be generating this kind of interference....

by -twitch-

Hi there, I was reading all these posts and I was wondering how it all turned out. Are there still issues or were they resolved and how?

It's always interesting to hear the way these things turn out.

I've had similar problems once. My efforts concluded that there was several different worms in the network that were not being detected by my anti-virus.

This discussion is closed to new replies.