Discussion Locked

Poor backup speed over a gigabit switch

0 helpful
10 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

Here is a difficult one:

I have a 3Com 3824 (24-port) gigabit switch. Seven of my nine Windows 2000 servers along with our developer's workstations are all plugged into it.

My issue is that backup jobs fail over the switch and/or never reach full speed. I am backing up to a Dell Powervault 725N that has dual Broadcom NICs.

My backup software is BackupExec 8.6 and I am using the Backup to Disk feature, which simply allows you to use shared folders on a server as backup devices. Here is an example of what exactly happens:

Server A = data source to be backed up
Server B = the backup server
Server C = the NAS box

I start a backup job from Server B and it starts out ok. After an unspecified amount of time the CPU on Server B server jumps up to 50 or 60%. It calms down in a minute or two. Next the CPU on the NAS (Server C) jumps up to about 60%.

Soon after that, the NAS becomes inaccessible over the network (using UNC) and the backup rate starts to drop. The backup job does not always fail at the same place, time, or amount of data.

We have replaced the CPU, RAM, motherboard on the NAS. We have replaced the switch. This has been an ongoing problem for me since Thanksgiving. I have updated NIC drivers, rebooted servers, etc, etc. I believe I may be missing something very fundamental. Everything has SP4 on it.

I have tested with CA's ARCServe and get similar results. I can almost assume the issue is with switching or NICs, or a setting on the NAS for MS Networking. Not sure

If I move the NAS to a 100MB/Sec switch and leave all the servers on gigabit, the backups run just fine. I simply cannot get the backup running when all the servers are on the gigabit switch. This was working before Thanksgiving but will not work now. There were no changes at Thanksgiving except for an extended power outage.

Why would backups be SLOWER with all the servers on the same switch?

Thanks in advance for any suggestions or help.

10 replies

by m_boyd

Have you looked at the Speed\Duplex settings on the NICs? I have seen instances where the switch will kill a port because the speed\duplex do not match at both ends, and there are too many Frame Collisions on the wire. If they are all auto\auto, try setting them all 100FULL and see if the result changes.

by tlandmesser

Poster rated this answer.
In gigabit there are no settings for hard-coding 1000MB/Sec connections, only 10 or 100. The only ports on my switch that I can control duplex for are the 1000 MB uplink ports that would go to another switch. Thank you for your input.

by tlandmesser

Also, there are no errors on the switch. No CRC issues, nothing to indicate that there is a problem with the switch itself. Almost seems OS related. Not sure

by robo_dev

Have you tried it with only one of the Broadcom nics plugged in? (some sort of spanning-tree problem or network loop created). Are the ports on auto-negotiate or nailed up at full duplex? I would look at the CRC error counters on the affected ports as this will indicate misnegotiation errors. Are you up to version 2.0 of the switch code?

by tlandmesser

Poster rated this answer.
You present very valid solutions. Currently there is a single Intel PRO 1000 GT Desktop adapter in the server. I thought it might have something to do with the teaming also so I disabled the on-board NICs.

All of the ports are set to auto as well as the NICs. There are no CRC errors at all. The replacement switch already had the 2.0 version loaded.

Thank you for your input

by rasilon

I would still try setting the NICs to manual (maybe during off shift) and see what effect it might have on the backup. If there is a significant improvement, then you should probably be talking to the vendor to see if there are drivers that will let you set the NIC to 1000. I've seen several experts recommend highly that all server NICs be set to Manual and *not* AUTO...

Hank Arnold

by mrtweak

Forget about forcing gigabit (Gb) hardware to Gb manually - there's no such thing.

Since you already mentioned the backup works with the 100Mb switch - does it work if the Gb switch & NICs are forced to 100Mb full-duplex? If not see (2) below. If it does I suspect the problem has nothing to do with networking hardware, but with source/destination server ability to sustain throughput. Go back to Gb settings and check other things: I suspect that the NAS drive/array can't handle all the input & is overwhelmed (also why it drops off the network), but does your NAS have any way to monitor disk usage/throughput and is it rated to sustain at least 200MB/sec (MB, not Mb) transfer? There is also a chance that the BackupExec servers are having trouble pre-processing/compressing the data (why CPU usage would be high) - try turning off software compression (assuming you don't have a NAS that has hardware compression) or reduce pre-processing? Cut pre-processing by simplifying the backup set. Try to find a big video file (700MB+ video file), make many copies in a folder, backup just that folder & see how far the backup gets. Also, a few settings on NICs can affect CPU/RAM usage on servers: enable "Offload Receive/Transmit TCP Checksum" or "Tx/Rx Checksum Offload" & in a separate test just toggle the Flow Control setting.

(2) Assuming you can't backup when forced to 100mb - the issues will be specific to the hardware combination and may require the manufacturers to contact each other to get it resolved. Replace everything outright - or try to force Dell to swap out hardware. In Nov. we forced Dell to swap a mobo with a Broadcom chipset for an Intel chipset & it resolved a problem with opslocks. I got the idea b/c of this thread where a guy has the opposite Intel/Broadcom problem: http://www.ndsengineers.com/printthread.php?t=150123. Don't know if you have a $$$ account that can "force" them...

by tlandmesser

Poster rated this answer.
MrT, I am awarding you these points and closing the question. When we force 100MB on the gig switch and server NICs, the backups still fail. It definately seems like some type of vendor issue. I have messed with the Offload RX/TX but to no avail.

I am putting in my two weeks on Friday. I have decided to accept an offer with another company. I will continue to work on this until my last day- it just bugs me that everything was working fine and all of a sudden it just "broke".

Thanks for your suggestions!

This discussion is closed to new replies.