I consider Blob Storage to be the gateway drug to Azure, because it’s a really easy way to get going with offsite backups.
One of the ways I’ve leveraged Blob Storage is with SQL Server backups. I even wrote a Sync and Restore command-line toolset to keep files and folders in sync between an on-premises file system and Blob Storage, because the tool provided by Microsoft, AzCopy, was just not up to the task.
As I wrote in a previous post, there were two main issues with AzCopy. The first was that the journal could get corrupted, and the
/y switch to resume or overwrite the copy process was not consistent enough to be reliable. The second issue was that there was no analogue to RoboCopy’s
To refresh your memory, RoboCopy (Robust Copy for Windows, built into Windows for a number of years) allows us to keep folders in sync between a source and destination using the mirror switch
/mir. In other words, if there are any changes to the source, these changes will be propagated to the destination. Files that no longer exist in the source will be deleted in the destination automatically. For managing file archives, this is hugely beneficial to help manage storage costs.
In the case of AzCopy, deleting files in the destination that were no longer in the source was fairly difficult to manage (hence my Sync tool above).
As of November 2018 however, the v10 preview of AzCopy is vastly improved. Firstly, it runs cross-platform on Windows, Linux and macOS (it is open-source, and appears to be written in Go). Secondly, it has more sensible command-line switches. Thirdly, and most importantly in my mind, it includes the much-awaited sync option. This has been a long time coming.
To synchronize a folder between on-premises (local) and off-site (Azure Blob Storage), we can issue a command like this:
azcopy sync "Z:\SQLData\Backup\INSTANCE" "https://myaccount.blob.core.windows.net/mycontainer/INSTANCE?sastoken" --recursive --force
The parameters for sync are as follows:
synckeyword, followed by
- the source folder
Z:\SQLData\Backup\INSTANCE(escaped with quotation marks)
- the destination container
https://myaccount.blob.core.windows.net/mycontainer/INSTANCE?sastoken(escaped with quotation marks)
--recursiveswitch, which tells azcopy to copy the entire structure below the source folder
--forceswitch, which will override any prompts (useful for automating this task)
Worth noting is that the Blob Storage container requires a SAS token for authentication, as opposed to the security key that was previously used. Generating a SAS token is done at the container level in the Azure Portal.
I have been using the v10 Preview of AzCopy at a customer site since November 2018, the day after it was released, and it has already saved them money in terms of maintenance and managing storage costs.
Feel free to leave your thoughts about AzCopy in the comments below.
Great article as i’ve been digging for more on AzCopy and using it with MSSQL
Great article, explained really well for a novice AZcopy user. I have a question, not SQL related. If during a AZcopy session the job gets cancelled half way through, I want to copy the folder again but NOT overwrite files already in BLOB that were copied on first pass, when prompted with “these files exist” do I select “overwrite disable” ?? Just need to copy what was missed, the last 50%.
Thanks a lot
Hi Neil, when I archive a SQL Server backup folder for example, I use
syncand it skips the files that already exist. Here’s what my command line looks like:
azcopy.exe sync "C:\SQLBackup" "https://my.blob.core.windows.net/container/?sv=<key>" --recursive --delete-destination true --cap-mbps 50
This is capping the speed to 50 Mbps, deleting any files in the destination that don’t exist on the source (useful for cleaning history), and recursive through subdirectories.
Hi Randolph, excellent article, but I can’t get it to work. My command looks like this:
azcopy.exe sync “C:\Temp\AzSyncTest” “https://scottsv2store.blob.core.windows.net/files/?sv=” –recursive
But get the following error:
RESPONSE Status: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
I created a SAS on the container “files”, so I’m not sure what I did wrong.
I’ve seen this be a problem if you have a proxy server somewhere between you and the Internet.
Great article, Randolph!
I need to mirror files from a local folder to an azure fileshare but it seems that this is not yet supported by azcopy. Do you have a suggestion maybe, how I could solve this task?
>azcopy.exe sync “c:\temp\storagetest\*” “https://.file.core.windows.net/datagis/testtarget1/testtarget2” –recursive
error parsing the input given by the user. Failed with error source ‘c:\temp\storagetest\*’ / destination ‘https://.file.core.windows.net/datagis/testtarget1/testtarget2?-REDACTED- combination ‘LocalFile’ not supported for sync command
This might help: https://github.com/Azure/azure-storage-azcopy/issues/830