Automating Amazon S3 backups on a Windows Server
by Alex Yumashev ·
Mar 18 2012
Our helpdesk app (the hosted version) is being used in about 170 companies. Users have uploaded almost 150k (150,000) files. We have logged about half a million tickets. From about 100k users.
Obviously, we need to backup all this data. And obviously, a consumer product like Dropbox or SugarSync is not an option. We needed something fast, reliable & scalable. We needed cloud storage. So we chose Amazon S3 and I decided to write this simple step-by-step guide for anyone interested in automating S3 backups on a Windows server:
If you don't already have an AWS account - create it here, it's free. Amazon's "free usage tier" on S3 gives you 5GB free storage from scratch, so after registering, sign in to your "AWS Management Console", select the "S3" tab and create one or more "buckets" on the left.
You will need security credentials to access your online storage from the server, so click your account name - "Security Credentials" - "Access Keys" and copy your Key ID and Secret.
"S3Sync" is a great free command-line application from SprightlySoft. It is .NET-based and even comes with the source codes. At the time of writing this post their website was down, so I published the tool on Google Docs here: S3Sync.zip.
The tool syncs a given folder with your S3 bucket. And the best part - unlike similar scripts and utilities it performs a "smart" differential sync that detects additions, deletions and file-modifications.
Create a batch file and paste this code into it:
The code above is pretty self-explanatory. Just replace the "xxxxxx" with your access codes from #2, "YOURBUCKETNAME" with the name of your S3 bucket, and "C:\inetpub\wwwroot" - with the folder you want to backup. Then create a scheduled task that runs the batch file every 24 hours, and you're all set.
Amazon gives you a 5 GB free storage during the first year. After the year ends the pricing is still very moderate. For instance, a 10GB storage will cost you about 60 cents a month which makes it a no-brainer.
Obviously, we need to backup all this data. And obviously, a consumer product like Dropbox or SugarSync is not an option. We needed something fast, reliable & scalable. We needed cloud storage. So we chose Amazon S3 and I decided to write this simple step-by-step guide for anyone interested in automating S3 backups on a Windows server:
1: Create an Amazon AWS account
If you don't already have an AWS account - create it here, it's free. Amazon's "free usage tier" on S3 gives you 5GB free storage from scratch, so after registering, sign in to your "AWS Management Console", select the "S3" tab and create one or more "buckets" on the left.
2: Get your access keys
You will need security credentials to access your online storage from the server, so click your account name - "Security Credentials" - "Access Keys" and copy your Key ID and Secret.
3: Download "S3Sync"
"S3Sync" is a great free command-line application from SprightlySoft. It is .NET-based and even comes with the source codes. At the time of writing this post their website was down, so I published the tool on Google Docs here: S3Sync.zip.
The tool syncs a given folder with your S3 bucket. And the best part - unlike similar scripts and utilities it performs a "smart" differential sync that detects additions, deletions and file-modifications.
4: Write a backup script
Create a batch file and paste this code into it:
S3Sync.exe -AWSAccessKeyId xxxxxxx -AWSSecretAccessKey xxxxxxx -SyncDirection upload -LocalFolderPath "C:\inetpub\wwwroot" -BucketName YOURBUCKETNAMEThe code above is pretty self-explanatory. Just replace the "xxxxxx" with your access codes from #2, "YOURBUCKETNAME" with the name of your S3 bucket, and "C:\inetpub\wwwroot" - with the folder you want to backup. Then create a scheduled task that runs the batch file every 24 hours, and you're all set.
5: Pricing
Amazon gives you a 5 GB free storage during the first year. After the year ends the pricing is still very moderate. For instance, a 10GB storage will cost you about 60 cents a month which makes it a no-brainer.