Questions

What can I do to speed up the copying of large files?

+
0 Votes
Locked

What can I do to speed up the copying of large files?

dennis.rhine
We are in the process of virtualizing many of the servers we use at the office and have encountered and issue that I didn't expect and cannot see a reason for. When backing up, or copying large files (over 1GB) the process goes so slow that it practically cannot be done. For example, I copied a couple of smaller files - about 2MB each - in a minute or so each. However, when I tried to copy a 3GB file it took several hours and then was only partially done. The impact on the network is that a backup that should take a few hours has exploded to over 60 hours.

Has anyone else experenced this? If so, what is the best way to fix it?
  • +
    1 Votes
    robo_dev

    I would guess you're talking about VMware ESX 4?

    I believe that the free ESXi has some limits on throughput...around 6MBPS.

    What are your approximate measured transfer rates?

    There are some great products from Veeam, FastSCP. (free versions too)

    Note that using plain old unsecure FTP is typically ten times faster than SCP.

    (Actually, I am pretty sure that FastSCP is actually FTP by another name)

    Of course, to/from another server puts great demands on the network, and VMware can be tricky to configure

    +
    1 Votes
    markp24

    Hi

    I have seen this on non vm desktops coping larges data files(ie pst files) to a network server, there nics were set to auto/auto so was the cisco switch,

    once i changed both the nic and the swithc port to 100/full duplex , everything copied much faster! You may want to try this on the host EXI (set it to 100 or gbit/ full duplex and the switch port, then see if that resovles it.

    +
    0 Votes
    chdchan

    I think there might be issues with network traffic or bandwidth that bogged down your speed. Even the commonest broadband router should deliver shorter time for file copy than yours. Perhaps, simply use USB drives/Dropbox (data transit) or shared folders (avoid duplication) to work around.

  • +
    1 Votes
    robo_dev

    I would guess you're talking about VMware ESX 4?

    I believe that the free ESXi has some limits on throughput...around 6MBPS.

    What are your approximate measured transfer rates?

    There are some great products from Veeam, FastSCP. (free versions too)

    Note that using plain old unsecure FTP is typically ten times faster than SCP.

    (Actually, I am pretty sure that FastSCP is actually FTP by another name)

    Of course, to/from another server puts great demands on the network, and VMware can be tricky to configure

    +
    1 Votes
    markp24

    Hi

    I have seen this on non vm desktops coping larges data files(ie pst files) to a network server, there nics were set to auto/auto so was the cisco switch,

    once i changed both the nic and the swithc port to 100/full duplex , everything copied much faster! You may want to try this on the host EXI (set it to 100 or gbit/ full duplex and the switch port, then see if that resovles it.

    +
    0 Votes
    chdchan

    I think there might be issues with network traffic or bandwidth that bogged down your speed. Even the commonest broadband router should deliver shorter time for file copy than yours. Perhaps, simply use USB drives/Dropbox (data transit) or shared folders (avoid duplication) to work around.