I have about 30GB of files which I recently decided to synchronize across the four computers I use (all running 32-bit Windows 7). Because I've gone almost completely paperless, I scan all the documents I can (better yet, I receive as many as possible in digital format, such as magazines and bank statements) and store them electronically. This consists of the following top-level folders on my D: drive.
- Books - E-books, technical PDFs, white papers, and other recreational/career documents
- Data - Financial data, receipts, and memorabilia. Most importantly, my "Work" folder which contains all my writing including the 580 page crime novel I spent almost a year on and which is now undergoing final revision.
- Installs - Install files such as Office, Adobe and VPN products
- My Pictures - My digital camera photos for the current calendar year (I archive the prior year's photos to a separate disk)
All told, this consists of almost 11,000 files in over 700 subfolders. It's a lot to manage, but it allows me to manage my job and my life from anywhere, since I'm not dependent on file cabinets or printed manuals. I'd probably need a moving van to haul around the hard copies of all this information!
I had used the free versions of Dropbox (3GB of space) and Google Drive (5GB of space) for some time, but I decided to pay for enough storage with one of these providers so I could keep all my files automatically updated on my systems. Previously I just kept my "Work" folder (about 1GB) in Dropbox and part of my "Data" folder in Google Drive, but I elected to integrate "Work" underneath this "Data" folder and go from there. My goal was one storage repository for everything; no hassles, manual updates or having to go here or there for what I needed.
It was a tough choice, since the one thing that always impressed me about Dropbox has been its rock-solid reliability. However, the factors that led me to choose Drive over Dropbox involved the following:
- Cost ($5 per month for 100GB of Drive space versus $10 per month for the same amount of Dropbox space)
- Integration with Google services, as I discussed a few months back
- The excellent support I've seen from Google in the past
As the saying goes, little did I know how soon I would be calling on that support.Setting up a paid Google Drive storage plan was easy; I accessed my Drive account at drive.google.com then clicked "Upgrade storage" in the lower left of my Chrome window. (Figure A)
I clicked "Choose this plan" under the 100GB option and after entering my payment information I was the proud owner of some hefty new Drive space. I got this accomplished before breakfast, so I dragged and dropped the remainder of my 30GB of files into the Google Drive folder on my PC, which is located on my D: drive.Then I observed the Google Drive icon in my system tray. (Figure B)
This icon shimmers when Drive is synchronizing files and shows as solid when the synchronization is complete. As I left for work the Drive icon was shimmering, so I figured it would upload my data to Drive and then copy it down to my work PC, which also has Drive installed on it.
The first sign of trouble
I got to work to find that only about 250MB of my data had synchronized from my home computer to my work system, within about forty minutes. That meant it would take over 2.5 hours per GB - more than three days to copy 30GB of data - to complete if the throughput remained the same.
Since I didn't want to use too much bandwidth at the office I came up with a clever solution (so I thought) - I would copy the files from my USB drive into my Google Drive folder on my work PC. Since most of these files were the exact same versions as the ones I was trying to copy down from the Internet, I figured Drive would recognize there was no need to sync these. This would, theoretically speaking, shave days off the synchronization process. From there on out it would be smooth sailing since only a small amount of my data is "dynamic" meaning that it changes frequently.
In a word: wrong!As I said before, my four main folders are called Books, Data, Installs, and My Pictures. Not long after I copied all my files into my work PC's Google Drive folder, I checked it and found Figure C.
Uh, why is there an extra folder called "Installs (1)"?
You can probably guess what happened - duplicate folders were starting to show up. Worse, duplicate files began streaming onto the hard drive - these appeared with (1), (2) or even (3) after the name.
Everything became a game of whack-a-mole after that. I spent a lot of time comparing my online Drive folders with those on my local systems and removing the duplicate objects, but they respawned as soon as I deleted them. I restarted the Drive application a few times to try to get sync working properly. Then I elected to start over; I moved all the data out of my Google Drive folders, deduplicated it using "Free Duplicate File Finder," and deleted it from my online Drive space.
I checked Google Drive help page but it didn't really seem to apply to my situation, so I contacted Google support. I've always had prompt service and good results from Google support, and so in short order I discussed the issue with one of their engineers. He advised me to move my Google Drive folder on my home PC from my D: drive to the same volume where the program was installed (in this case the C: drive). Explaining that sync problems can occur if the Drive repository is on a different volume, he also recommended slowly re-adding my files to my Google Drive folder so as to not choke it with 30GB of data at once.
I gave this a try, spending the weekend carefully feeding my data into my Google Drive folder. Slowly the folder and files began showing up in my online Drive account, and when all seemed to be uploaded I checked my work PC and things looked good. I did have to restart the Drive app a few times since the sync process appeared to hang (it would say it had completed 39/2054 files for instance, but then go no further), but finally it reported "Sync Complete" on my home and work PC. Those red X's next to the folders in the screenshot above (which indicate not all files have synchronized) went away. Situation normal, I presumed.
The return of the villain
You know those horror movies where the bad guy is supposed to be dead and the clueless kids poke him with their foot to see if he's still breathing? Yeah, something like that happened here.
I got up the next morning and tried to open a shortcut to a crucial file in my "Work" folder, which as I mentioned is a subfolder of "Data." The shortcut didn't open.
"How weird," I thought, and opened Windows explorer to examine the contents of "Data." My Work folder had completely disappeared. Cue the scary bass music.
I have a nightly scheduled task which copies all my files onto an external drive for backup purposes. This is different from synchronizing the files, since it never removes anything from the target location. I do this so that if something is accidentally deleted from my hard drive the unwanted removal won't replicate to my external drive, because that would defeat my backup scheme. I can also get stuff I erased then changed my mind about deleting, such as drivers that I might want to reinstall.
I was able to recover my files from backup, and I observed that my online Drive account did show the "Work" subfolder, so not all was lost. I copied the "Work" folder back under "Data." After a while the Google Drive app reported "Sync complete." The "Work" folder never appeared on my office PC, despite the fact Google Drive also reported "Sync complete" on that computer.
Since the Work folder is so critical to me I moved it to my Dropbox account until I could sort things out. What makes me nervous here is not losing stuff I use every day (since I'll notice the absence right away), but missing files which I might not realize are gone for several days or even weeks.
I left the other folders as is. Then I recorded the size and number of items in my top-level folders to compare these between my two systems and track how Google Drive synchronizes items. I've been slowly adding and removing test files to my Google Drive folders and checking to see if these changes are properly replicated. So far all is working as expected, though I've probably changed 25 or less items.
The jury is still out
I haven't given up on using Google Drive just yet, though I have to admit that I'm not willing to take any risks with the data I rely on. I'm awaiting further advice from support on how to properly troubleshoot this problem. There is a sync log for Google Drive, which on Windows 7 systems is located under %LOCALAPPDATA%\Google\Drive\sync_log.log. My log file is 88MB and thus far I've found no explanation in it for the removal of the "Work" folder. There's too much data in there to wander through it without a guide. A Google product forum discusses the issue which shows I'm not alone.
There are a few good takeaways from this experience
Adding 30GB of data to my Google Drive folder all at once was an obvious mistake; lesson learned. It was like trying to drive a school bus down the sidewalk. I'd previously used 4GB of data with Google Drive and had no discernible problems.
I have a ton of data. It's not just the 30GB involved, but the 11,000 files as well. I'm something of a digital packrat, I admit. I have utilities dating back to Windows 95. I think part of the problem here involved the sheer amount of objects. Zipping up many of the older items I don't often access would probably help ensure smoother synchronization, since there would be less material to comb through when analyzing changes.
This problem may not be native to Google Drive. I'm currently experimenting with 50GB of free space from Tresorit and the results have been sluggish as well with such a high data volume, though I've not yet found any discrepancies. It would be interesting to see if similar experiences also occur with Dropbox.
The Google Drive app seems to have performance issues when it's busy for too long (at least in my experience) - having to restart it periodically obviously didn't help with the synchronization process. Again, this is possibly tied into too much data flowing at once.
It would have been a good idea NOT to have the Google Drive program running on another system until all my files were successfully synchronized from my home PC to Google servers. I'm sure this added to the confusion since the work PC was also trying to pull down the changes.
Always have multiple copies of your data, and DON'T synchronize everything to another location; use a standard copy method so you can get back any files erroneously deleted. Schedule the backups so they don't kick off as soon as changes are made, but rather run periodically so you have time to recover.
I'll update you on this problem in the Comments section of this article once I find out more - and if you've seen similar issues please let me know.
Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.