Multipart upload to Amazon Simple Storage Service with CloudBerry Lab S3 Explorer

Sometimes getting data to a cloud storage provider is the biggest obstacle to embracing cloud storage. IT pro Rick Vanover shows how to use a new feature to upload data to the S3 cloud.

When it comes to cloud storage, the end-user has a lot of options. One of the more popular storage destinations is the Amazon Web Services (AWS) Simple Storage Service (S3) cloud. The S3 cloud is one of the most popular storage options in cloud technologies today. It is simple, scalable and affordable.

The issue a lot of organizations may have is the upload to a storage cloud such as S3. Because S3 is accessed via REST and SOAP over the Internet, issues are the norm with uploads. This can cause large downloads to have to stop and start again, adding frustration as well as bandwidth transfer cost to the upload process.

Recently, the CloudBerry S3 Explorer PRO product introduced the multipart upload feature. Multipart upload is rather straightforward; it breaks large files up into chunks in user-specified sizes. The multipart configuration upload interface is shown in Figure A below: Figure A

Using multipart upload allows the source file(s) to be uploaded in chunks as small as 5 MB and as large as 5 GB. Should the upload need to be paused, only the job is resumed at the current chunk, rather than a full restart of the transfer.

When chunks are being uploaded, they are not visible in CloudBerry explorer. If another product is used to view the contents of the S3 bucket or folder that is receiving the multipart upload, you can see how CloudBerry arranges the transfer in Figure B below (click thumbnail to enlarge): Figure B

It wouldn’t be a good idea to interfere with the chunks as they are being moved, as CloudBerry S3 Explorer PRO needs everything to be a certain way to re-assemble the chunks when the transfers are complete. More information on the multipart upload feature can be found on the CloudBerry blog.

The chunking process of multi-part uploads will not add efficiencies to a file that has grown, however. A good example would be a ZIP file that was transferred to S3; and then that file has additional content added to it. The ZIP file would need to be fully transferred again, not just the new chunks.

The multipart upload option for CloudBerry S3 Explorer PRO is a nice touch to making big transfers less painful. This is especially the case in my home lab, which has limited upload bandwidth from my Internet service provider. How have you managed large uploads to storage clouds? Share your comments below.