AzCopy finally gets a sync option, and all the world rejoices

I consider Blob Storage to be the gateway drug to Azure, because it’s a really easy way to get going with offsite backups.

One of the ways I’ve leveraged Blob Storage is with SQL Server backups. I even wrote a Sync and Restore command-line toolset to keep files and folders in sync between an on-premises file system and Blob Storage, because the tool provided by Microsoft, AzCopy, was just not up to the task.

As I wrote in a previous post, there were two main issues with AzCopy. The first was that the journal could get corrupted, and the /y switch to resume or overwrite the copy process was not consistent enough to be reliable. The second issue was that there was no analogue to RoboCopy’s /mir switch.

To refresh your memory, RoboCopy (Robust Copy for Windows, built into Windows for a number of years) allows us to keep folders in sync between a source and destination using the mirror switch /mir. In other words, if there are any changes to the source, these changes will be propagated to the destination. Files that no longer exist in the source will be deleted in the destination automatically. For managing file archives, this is hugely beneficial to help manage storage costs.

In the case of AzCopy, deleting files in the destination that were no longer in the source was fairly difficult to manage (hence my Sync tool above).

As of November 2018 however, the v10 preview of AzCopy is vastly improved. Firstly, it runs cross-platform on Windows, Linux and macOS (it is open-source, and appears to be written in Go). Secondly, it has more sensible command-line switches. Thirdly, and most importantly in my mind, it includes the much-awaited sync option. This has been a long time coming.

To synchronize a folder between on-premises (local) and off-site (Azure Blob Storage), we can issue a command like this:

azcopy sync "Z:\SQLData\Backup\INSTANCE" "https://myaccount.blob.core.windows.net/mycontainer/INSTANCE?sastoken" --recursive --force

The parameters for sync are as follows:

• the sync keyword, followed by
• the source folder Z:\SQLData\Backup\INSTANCE (escaped with quotation marks)
• the destination container https://myaccount.blob.core.windows.net/mycontainer/INSTANCE?sastoken (escaped with quotation marks)
• the --recursive switch, which tells azcopy to copy the entire structure below the source folder
• the --force switch, which will override any prompts (useful for automating this task)

Worth noting is that the Blob Storage container requires a SAS token for authentication, as opposed to the security key that was previously used. Generating a SAS token is done at the container level in the Azure Portal.

I have been using the v10 Preview of AzCopy at a customer site since November 2018, the day after it was released, and it has already saved them money in terms of maintenance and managing storage costs.

Feel free to leave your thoughts about AzCopy in the comments below.