Backing up my server to Amazon S3

October 3, 2006 | In Uncategorized |

I have tried a few different backup strategies for my Linux servers. Before I used rdiff-backup to mirror directories to a backup server. I have also tried a home built shell script that builds tar archives, encrypts them with gpg and upload them to a backup area with ftp.

This time I feel I have a better solution. I use the tool duplicity and the Amazon S3 web service.

It wcould be easier to setup. Duplicity only recently got support for S3 so I had to get the code from CVS. Then I needed to hunt down the BitBucket Python library to get the S3 connection working. But now it works really well.

Duplicity uses the same library as rdiff-backup so it will only transfer the data that really has changed. So if a file grew from 2 to 3 Mb, only the new Mb will be transfered.

S3, for you who have been living without Internet the last six months, is a storage service from Amazon. You register with them and receive an user id and password (but Amazon have some strange names for them). Then you use a web service to send and recieve data into “buckets”. The buckets gets stored on Amazon’s big storage network. The data is encrypted with your key, but Duplicity can also encrypt or sign the data before you send it.

Amazon charges for the data you store and the amount of data you transfer. And the prices are really competitive. 20 Gb with 2 Gb monthly transfer will cost you USD3.40. I think we will end up closer to $15 after the first six months. I intend to burn DVD:s every second month or so and store them somewhere safe. No need to keep the whole history on S3 - just the current stuff.

So with everything working I write a list of directories I want to backup. Duplicity has really powerful include/exclude syntax so you can be really selective if you want to save money. Then I put it into the crontab and let it do its magic every night.

Don’t forget to test recovery now and then so you know that the backup is working.

7 Comments

TrackBack URI

  1. could you post your script that you use to backup your files to s3? i couldn’t find any good documentation for duplicity and s3 and i’m not yet a linux guru.
    but this seems to be the cheapest solution to backup my server.

    Comment by nick — November 1, 2006 #

  2. Actually it’s very simple once you have duplicity and BitBucket installed:

    Here is how I run it:

    #!/bin/sh
    export S3KEY=[key]
    export S3SECRET=[secret]
    /usr/local/share/duplicity/duplicity-bin --include-filelist="backuplist.txt" / s3+http://my-bucket

    Good luck!

    Comment by marvi — November 1, 2006 #

  3. […] The other option is a somewhat DIY solution (which on hindsight resembles this one): […]

    Pingback by The Life and Times of Michael Shadle » Blog Archive » The quest for the end-all backup solution — December 31, 2006 #

  4. […] http://labora.harnvi.net/?p=34 […]

    Pingback by Docunext Tech Stuff Archive » Duplicity to dreamhost and Amazon S3 — January 29, 2007 #

  5. The error message in the last pingback seems to be about a missing library. I have librsync-0.9.7 installed.

    Comment by marvi — January 29, 2007 #

  6. […] Then create a bash script to put it all together. I used the script provided by Labora. #!/bin/sh export S3KEY=[key] export S3SECRET=[secret] /usr/local/share/duplicity/duplicity-bin –include-filelist=”backuplist.txt” / s3+http://my-bucket […]

    Pingback by Docunext Tech Stuff Archive » S3 Duplicity with Debian — January 29, 2007 #

  7. Good post! I’ve been looking for something to back up my server remotely.

    FYI the bitbucket link you had seems to be broken. I think the URL is now

    http://code.google.com/p/bitbucket-s3/

    - Ben

    Comment by Ben Strackany — April 25, 2007 #

Sorry, the comment form is closed at this time.

Entries and comments feeds. Valid XHTML and CSS. ^Top^