Windows

Use Robocopy's multi-threaded feature to quickly back up your data in Windows 7

Greg Shultz shows you how to take advantage of Robocopy's multi-threaded copy feature to create an exact mirrored duplicate of your user profile folder.

Soon after Windows Vista came out, I discovered that Microsoft had updated the venerable Robocopy with a bunch of extra features and made it a regular part of the Windows operating system. Of course, when Windows 7 came out I immediately examined the newest version of Robocopy and discovered that Microsoft had again updated Robocopy with a single, but very powerful, new feature -- the ability to perform a multi-threaded copy option. More specifically, with multi-threaded capabilities Robocopy can now simultaneously copy multiple files in parallel, which will result in very fast backup operations.

In this edition of the Windows Desktop Report, I'll show you how to take advantage of Robocopy's features to create an exact mirrored duplicate of all the pertinent data files in your user profile folder (C:\Users\YourName). As I do, I'll show you how to use the new multi-threaded copy feature.

This blog post is also available in PDF format in a TechRepublic download.

Why use Robocopy?

If you're conscientious about the safety of your data, chances are good that you are already using Windows 7's Backup and Restore tool to create an image file of your hard disk as well as to back up your data files on a regular schedule. However, you may like to have an additional copy of your data files just to be on the safe side. While you can easily do so by copying your files and folders to an external hard drive via drag and drop, that can be a tedious operation.

Fortunately, Robocopy provides more than 80 different command-line switches that will allow you to create a powerful data backup operation. To help you get a leg up, I have delved into Robocopy 's command-line switches and have developed a nice little script that you can use to create an exact mirrored duplicate of all the data files in your user profile folder (C:\Users\YourName). Let's take a closer look.

Constructing the command line

For my example, I'm going to back up the data file contents of my user profile folder, C:\Users\Greg Shultz, to a folder named TheBackup on an external hard disk that is assigned drive letter G. (You'll, of course, substitute the names and paths with your own.) As such, my command will begin with:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup"

Now, I want to back up every folder in the source, even any empty folders, as they may be placeholders for future data. I also don't want to have files on the backup that I deleted from my hard disk. While I can use the /S and /PURGE switches to accomplish my goal, I can use the /MIR switch to accomplish both of these tasks with one switch. As such, my command is now:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup" /MIR

The C:\Users\Greg Shultz folder contains several hidden, system files and folders that I don't want or need to back up. For example, I don't need to back up the NTUSER.DAT file nor do I need to back up the contents of the AppData folder. In addition, the C:\Users\Greg Shultz folder contains a host of junction points that I don't need to back up.

Windows 7 uses junction points to link various operating system folders to the user profile folder. For example, the Cookies folder and the SendTo folder are linked to the user profile folder via junction points.

I'll use the /XA:SH switch to exclude the hidden, system files, and I can use the /XD AppData to exclude the entire AppData folder. I'll then use the /XJD to exclude all the junction points. As such, my command is now:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup" /MIR /XA:SH /XD AppData /XJD

One of Robocopy's most useful features comes in to play when it encounters a file that is in use. When it does, Robocopy will stop and wait for that file to close so that it can continue with the copy operation. It will retry to copy the file every 30 seconds. The default number of retries is one million (no joke!). As this will most like prevent the backup operation from ever completing, you should reset it to a reasonable number.

To change the number of retries, you'll use the /R switch, and to change the wait time between retries, you'll use the /W switch. I chose 5 retries with a 15-second wait time. That way after a reasonable number of retries and wait period, Robocopy will move on. As such, my command is now:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup" /MIR /XA:SH /XD AppData /XJD /R:5 /W:15

At this point, I am ready to add the new multi-threaded switch: /MT[:n], where n is a number from 1 to 128 and indicates the number of threads to be used. Keep in mind that n is optional and that by default, the /MT switch will use 8 threads. I'll use 32 threads in my example, as I found it to be a good starting point. (Note that the multi-threaded option is not compatible with the /IPG and /EFSRAW switches.) At this point, my command is now:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup" /MIR /XA:SH /XD AppData /XJD /R:5 /W:15 /MT:32

Like all command-line tools, Robocopy keeps you apprised of the status of the operation right in the Command Prompt window. However, chances are that you'll want to customize and record that feedback in a log file. I like to have the whole picture, so I'll use the /V switch. However, I really don't need to know the percentage progress of each file copy, so I'll use the /NP switch. To create my log file, I'll use the /LOG switch, which will overwrite the existing log file each time. Now, my command is:

Robocopy "C:\Users\Greg Shultz" "G:\TheBackup" /MIR /XA:SH /XD AppData /XJD /R:5 /W:15 /MT:32 /V /NP /LOG:Backup.log

Creating and using your script

Now that you know how the script works and the necessary switches, you can launch Notepad, type the command, and save the file as RobocopyBackup.cmd. To make sure that the script and open log file don't interfere with the backup, I created a folder in the root directory called BackupTool (C:\BackupTool) and saved the script there.

You'll find the log file in the same directory as the script after each backup operation. Keep in mind that while the log file is a simple text file, it can be larger than Notepad can handle. As such, you may want to use Wordpad or another word processor to open and view the log file.

Now, anytime that you want to make an extra backup, you can just double-click on the RobocopyBackup.cmd to launch it. When it is done, you can examine the Backup.log file. You can also use the Task Scheduler to automatically run your RobocopyBackup.cmd on a regular basis if you want.

What's your take?

Have you used Windows 7's version of Robocopy? If so, what's been your experience? Would you add any additional switches to the script that I presented? As always, if you have comments or information to share about this topic, please take a moment to drop by the TechRepublic Community Forums and let us hear from you.

Stay on top of the latest Microsoft Windows tips and tricks with TechRepublic's Windows Desktop newsletter, delivered every Monday and Thursday. Automatically sign up today!

About

Greg Shultz is a freelance Technical Writer. Previously, he has worked as Documentation Specialist in the software industry, a Technical Support Specialist in educational industry, and a Technical Journalist in the computer publishing industry.

21 comments
jayden82821
jayden82821

Long path tool can helpful on this situation. thanksĀ 

jayden82821
jayden82821

Long path tool can helpful on this situation. Thanks

jim
jim

So... I have been trying desparetly to find a way to copy my windows 7 profile to a new location on the same computer. It's loading really slow and I think if I copy the info to a new profile it might clean it up and have it load faster (that thinking alone could be faulty). So it looks like I could use your robocopy c:\jim c:\newjim /E /copyall /XF ntuser /r:1 /w:3

keith.v.irvin
keith.v.irvin

Just curious of the reason for excluding the AppData folder? For example, all of my email is at: C:\Users\username\AppData\Local\Microsoft\Windows Live Mail And my IE QuickLaunch toolbar is at: C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer\Quick Launch

msmalik48
msmalik48

I'll rather use a more powerful free softawere "XXCOPY", instead.

8string
8string

It's always been a great tool, and now it's even better. Thanks for the overview, I'd forgotten about it lately.

gtrt4y
gtrt4y

Yes i use it as you show. Would have liked this earlier. I started with the simple as you show and progressively added to refine with the exception switches.

pjboyles
pjboyles

You may want to use /E rather than /MIR. It preservers data you have deleted from your machine in the archive as well as directory structures. You never know when that historical data may become useful. Adding /XX has it ignore extra files in the destination. And if you want to log and see what is happening /TEE. Note that /TEE does slow copying of many files. And for those pesky open files /B may come to your rescue if they can be opened in backup mode. I don't recommend /Z as it adds a tremendous amount of overhead. Copying to a remote share that may have some network issues it becomes a necessity for large files. Lastly /REG to set your Retry and Wait times as default. (If you have a setup script for your profile because you test a lot, add "robocopy /R:5 /W:15 /REG" to it and have it configured. I do "net use /p:no" as well.) A final thought on /MT, watch increasing this when copying to a network share (especially one not on the local segment).

Mark W. Kaelin
Mark W. Kaelin

Have you used Windows 7's version of Robocopy? If so, what's been your experience? Would you add any additional switches to the script that Greg presented?

Greg Shultz
Greg Shultz

...I designed the Robocopy backup script as a supplement to the standard Windows backups to allow you to simply make additional copies of user created data files. I excluded the AppData folder because the majority of its content is configuration data for applications. Besides, Windows 7?s Backup and Restore tool will capture all all of the data stored in the AppData folder. However, the exclusion is just my personnel preference. If you wish to back up the AppData folder, just exclude the /XD AppData switch from the command line.

gkeramidas
gkeramidas

remove certain files from folders, that are more than 30 days old.

Justin James
Justin James

... is around the open files issue. What about the person who leaves Outlook open? That PST is arguably their most important file. It also means that I can't use this on a server, say, to backup the running VMs. Too bad it doesn't use VSS built in. :( J.Ja

jim
jim

Of course I created an additional user first! It was seeing how that loaded that made me suspicious in the first place. No - I was not aware of that article. I will review it over the weekend. Thanks for the lead.

Greg Shultz
Greg Shultz

...I don't really think that fact that it has a GUI means that Robocopy is obsoloete. Command line tools are still relevant these days and Robocopy works perfectly. Besides if Robocopy were indeed obsolete, I would think that RichCopy would have been a part of Windows 7.

jphoeke
jphoeke

I just created a batch file to run to delete BAK files that are older than 30 days using the Forfiles command. Forfiles /p "E:\Network\GenData" /s /m *.bak /d -30 /c "Cmd /C del @file"

JamesS_R
JamesS_R

I've been using it for 3years to shift database log files and document repositories onto warm standby servers. Very easy to set up, reliable and a good granularity of control when needed.

Gis Bun
Gis Bun

RichCopy came out just before Win 7 was released. Maybe they didn't have time to insert it. Who knows. While RoboCopy is command line RichCopy has native 64-bit support and you can also save job settings.

Editor's Picks