Discussion Locked

High density rack wiring - best practices?

0 helpful
20 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

We're seeing our datacenter migrate from 5U and 7U servers to 1U and 2U servers. With the older servers - things were fairly simple, but we're starting to cram 12-15 servers into a 42U rack (with room for UPS's, monitor, keyboard drawer, KVM, etc...).

Obviously this is creating a bit of a wiring problem and could use some ideas on how to better manage wiring, especially with long and thick power cords and KVM wiring. The 2U rack servers we're buying have an articulated arm for better cable management but these things are a pain to deal with and get around.

Is there a good best practices guide out there? I'm not 100% sold on the network KVM's because of performance issues and expense?

TIA.

20 replies

by danmcl

I havent seen many best practise guides, only photos of what and what not to do!
I find that either reusable nylon zip-ties or velcro ties are very useful, I prefer the velcro ties but if expense is an issue with your purchasing department then the releasable nylon zip ties are just as good, just a bit harder to work with later on.

I put in a 1u cable management tray about halfway up the rack and bring ethernet to the top and power to the bottom or sides depending on where the distribution units are.

I keep the power and ethernet on separate sides of the rack and have them tied on.

I also leave a bundle of cable tied up with the velcro ties(power and ethernet) for when I need to slide the server out, I see a lot of installations where people forget to do this and then end up pulling the plug on a server non-intentionally (unintentionally???)

edit: FWIW: The server rack in question ^^^ has 26 servers (mix of 1u and 2u) as well as a bundle of structured cabling going back to a separate rack for patching into a Cisco Catalyst

I inherited a very badly installed rack that was in a standard office (no A/C etc.), the room was regularly at 40degC plus. Sorting the cables out at the back improved airflow significantly and I saw a temp drop of 15degC!

by now left tr

I would say similar having just gone through the same process in April. It is well worth the putting in the effort at the time.

by rzrwire

A couple small things that make a BIG difference... buy smaller power cables, avail via CDW etc, you can get them as short as 3 ft, and a lot thinner than std power cables. This saves a lot of space and hassle. Also IP KVMS (we use the Raritan models) save a TON of space, especially compared to those long, heavy, awkward, standard KVM cables.

by rurick

One minor thing that we found helped. We moved to Avocent KVM switches. They dont use the traditional KVM cables. they use standard cat5 patch cables, which of course can easily be cut to the exact length you need. Just buy the KVM and a dongle for each server, and run cat5 cables between the two (only up to 30 feet or meters, cant remember which but plenty of room to run between racks) and you are set to go. Also, these KVMs arent but several hundred more than a standard KVM fully loaded. WELL worth the cost.

Dont skimp on double sided velcro wraps and zip ties. they are a lifesaver.

Also, custom cut your power cords. pickup a box of power plugs from the hardware store. figure out how long of a power cord you need, and custom make your power cords to remove excess length. (make sure you are qualified of course)

We went from rats nest to neat freak after all of the above.

by seckel109

I whole heartedly agree with rurick. We evaluated a number of KVM switches a couple of years ago. And the Avocent by far was the best.Expensive yes. Unobtrusive indeed.Support? Outstanding. This device along with on-site fabricated cables (power and data) is just the ticket to make your data center a keeper.

by dave.bruner

I have to agree with the Avocent KVMs. We also use color coded cat-5 cables to distinguish the KVM from the network. This helps and adds some color to the cabling!

by why me worry?

Try Raritan KVM switches, as the management interface is built into the KVM unit itself and does not require a separate Windows server to manage any of them. I've used Raritan for years and they are by far the best compared to HP KVM switches and those from Avocent.

by loadedmind

I would think that before anyone would post something so dramatic a statement as this would be better informed before doing so. Avocent is every bit as compatible with linux, Solaris, and Unix than Windows. In fact, the software will run on Solaris, Suse, RedHat, then Windows... Where do you get your information anyhow?!

by tomw

It has been the answer for us. 65 server down to less than half a rack. Cabling is not an issue. The saving on hardware maintenance, power, cooling, etc make this an easy one to pitch.

by bradl

We have 45 virtual servers on 4 physical 2U machines - 8 NIC ports and 3 power cables each. iSCSI SAN and LAN cables are color coded and wrapped in velcro strips tied to one side of cabinet, power cables wrapped and tied to the other side plugged into a 3 phase APC power strip mounted vertically in the cabinet, with each power supply of a server plugged into a differnet phase. We are at 30% capacity for VMs so we can grow quite a bit without adding any cables, UPS, cooling, etc. Saving serious dollars and adding agility - and the VMs are faster and more reliable than comparable hardware - go figure.

by pmatvey84

Virtualization is a great way to go to simplify infrastructure; we are hoping to implement it in our enterprise. What flavor did you end up using (ex. ESX, VM Server, Windows Vitual Server)?

by melbert09

Here are a couple of ways of saving space and making things look clean:

1. Custom cut all your CAT5/CAT6 cable. Takes a while but in the end it usually looks good. Plus you wont have cables stuffed under your servers in the rack.

2. USE the cable management arms that came with your servers. Make sure you properly put Power leads and Network lines in there and properly tied. Makes pulling the servers out of the Rack much easier.

3. Depending on the color of your racks, use "color Corrdinated" Zip Ties and Black Velcro straps to tie cabling to the side.

4. Use a Raritan IP KVM. Even if you dont use the IP part and just used the Monitor and Keyboard and Mouse directly connected. Its worth it, for space and cleanliness. Only uses CAT 5 cabling.

5. If you can afford to, color code your cabling for different networks this makes tracing cables soooo much easier. And it looks nice as well.

by cyberjunkie21

We're facing the same boat as you. Unfortunately our racks were planned without forethought So the cableing is a nightmare. One of our biggest hurdles is power now, we're reaching capacity quickly and are going to have to add more service to the building. We have the Raritan KVMs as well and it's quite awesome having it go over CAT5.

by angry_white_male

I never thought about shortening the power cables - fairly easy task. My boss will surely balk at the expense of new CAT5 KVM's... but that's what the next budget year is for.

Thanks everyone.

by ceswanson

We have a HP 10000 series 42U rack and we use HP servers in sizes ranging from 1U to 5U. We also use the HP IP KVM because of smaller cables and we can put 16 servers on 1 KVM instead of requiring 2 of the traditional 8 port KVMs with the huge cables.

Get vertical PDU's so you can use the shortest possible power cables. Like was stated in another post, we ordered 3ft and 6ft power cables instead of using the 9-12ft ones that came with the servers. You do get a lesser gauge of wiring (oem is 14, aftermarket is 16) but they seem to work fine. Using vertical PDU's and short power cables, I elimintate the routing of power cables down the rack.

On servers 2U and more, I use the cable management arms that came with the HP servers. They work great, and even lift up and pull back far enough to allow me to work on the back of the computer or get to the inner section of the arm.

I use a patch panel near the top of the rack to help manage network cables. I then custom make the cables to length, including any length that gets routed through the cable management arms.

Power wires loop out and to the plugs, KVM cables route to one side and up to the KVM switch, and network cables route to the other side and up to the patch panel.

So inside the rack, all the wires are routed to the sides and up, zip tied together. The bundle of network cables comes out the top and into the ceiling all nice and tight, and the power cord to the wall comes out the bottom. Very clean.

by dzenizo

I'd recommend to put the server's network switches on each rack instead of in the wiring rack as recommended by the usual "structurated cable" guidelines; In this way cable runs inside the rack are cleaner.
We usually put two 1U 24 port gigabit switches for redundancy at the middle of the rack facing backwards (they usually fit behind the slide-out console), run one cable from each of the two switches to each server for redundancy/load-sharing and then run as many cables from the switches to the central switch as needed for trunk aggregation. This has saved us a LOT of cost on labor and as a bonus we get cleaner cabling. BTW, we have about 600 servers wired this way.

One additional word about the cable management arms (they work great, BTW). If you standardize (highly recommended), then you can pre-assemble the cables in the arms BEFORE mounting the arms, this will save you a lot of time and effort. You can do that by making a "sample" arm, starting from the cables that go to the server and go then to the base of the arms, since that's the variable length that has to be managed.
Once you have the right measurements for one arm, you just copy this sample, install the arm and route the cables from the base to te switches, PDUs, KVMs, SAN switches, etc.

A final thougth about rack placement: Avoid the "paranoia" of packing the racks side to side; leave a manageable sppace between them. In our case, an inch or two between racks doesn't make a difference in space requirements and this helps with cooling and future needs, like removing a side pannel to better route cables, or trying to fit a new rack that is a LITTLE wider than the standard.

by why me worry?

I'm not a big fan of blade or "pizza box" servers, because although they are designed to squeeze more server into less space, they generate more heat and fail quicker due to thermal shutdown. If I were you, I'd worry more about the added cooling capacity needed for the data center and less about managing wiring, not to say that cable management isn't an issue, but systems that constantly overheat will be a bigger problem than keeping the wiring all nice and neat.

by angry_white_male

Last year we installed a 20-ton AC unit in a room with only 30 servers... believe me - cooling is not a problem in there!

by dzenizo

From my experience, I can tell you that good site cooling doesn't necesarly means good rack mounted server cooling!

I've seen rack "behinds" so crammed with cables that they obstruct air flow, and the servers were shutting down because of overheating, eventhough the rrom was at a constant freezing temperature. Also, I witnessed one site that had backup and operating procedures sheets pasted in front of the servers, using the suction force to hold down the papers!
Even in these days of ISO and SOX certifications, there are many "professional" sites that don't monitor servers internal environmental variables.
So good air flow, aided by good wiring practices, IS a big factor

by sgregory217

If you have a raised floor with cooling under the floor, I recommend wrightline's TOC cabinets that have air manifold doors for intake and exhaust. They have proven pretty efficient for us(providing cooling for 36-38 1U dell servers. We also use Server Technologies'84 receptacle PDU with custom length power cords and run all our servers at 220v to reduce the current draw. We've also found that the Sun servers that run Windows OS run cooler and take less juice. We have a 48 port patch panel at the top of each rack inside and run data and kvmoip cabling opposite the power cords. The custom length power cords have made a huge difference in cable management within the cabinet.

TOC info
http://www.wrightline.com/productDetail.asp?ProductID=7&ProductCategoryID=8&SubCategoryID=0

PDU info
http://servertech.com/products/3PhasePowerDistributionMetering/SmartCDU84-DD/

This discussion is closed to new replies.