http://grysz.com/2011/04/04/configure-amazon-s3-backup-with-backupninja-and-duplicity/

Data backup plan:

* pick desired directories with your data you want to backup
* you may backup system directories of your choice (in principle /etc /root /home /usr/local /srv etc.)
* optionally you may backup MySQL database

Used tools (tested on Debian Jessie and Ubuntu 12.04):

* BackupNinja 0.9.10 – for MySQL database backup and backup orchestration
* Duplicity 0.6.23 – filesystem backup
* duplicity-backup – wrapper shell script performing a backup using duplicity

Step 1. Install the tools

apt-get install backupninja duplicity python-paramiko python-lockfile dialog

! Do NOT install Duplicity 0.6.18 from Ubuntu 12.04 default repository, this version has buggy ssh client paramiko library !

Step 2. Configure Backupninja schedule

Check Backupninja configuration (/etc/backupninja.conf) if it suits your needs. The most important parameter is when it controls the backup schedule. I left it with the default value (everyday at 01:00). If you want to get backup report, fill at least `reportemail` parameter value.

Backupninja run every hour from cron job /etc/cron.d/backupninja to launch scheduled backup tasks.

Step 3. Backup MySQL database (optional)

Create a Backupninja backup action for MySQL database dump. The easiest way is to start “ninjahelper” tool and go through create a new backup action/mysql database backup wizard:

Directory to store backups: /var/backups/mysql
Backup all of databases: Yes
MySQL authentication method: password
Create a backup using mysqldump: sqldump
Compress the SQL output files: compress

Ninjahelper creates the action configuration in the /etc/backup.d/20.mysql file.

Perform a test:

backupninja --now --debug

Check if there has been no error messages and if the dump has been created in /var/backups/mysql/sqldump.

Step 4. Backup list of installed packages

echo "aptitude -F %p search \"?and(?installed,?not(?automatic))\" > /var/backups/pkglist.txt" > /etc/backup.d/25.pkglist.sh

The numerical prefix must be lower than last backup action.

Set the owner, group and permissions of new action script:

chown root:root /etc/backup.d/25.pkglist.sh
chmod 700 /etc/backup.d/25.pkglist.sh

Step 5. Exchange SSH key with backup server

From the server, which should be backed up run following commands as `root` user.

* Check if SSH keys already exists: ls ~/.ssh/id_rsa*
* If it doesn’t exist yet, generate them running command: ssh-keygen
* Install root user public key to backup server: ssh-copy-id ue]tod[kinterivznull]ta[xiaf
* Add known host key to known_hosts running: ssh-keyscan -t rsa zviretnik.eu>>~/.ssh/known_hosts

Step 6. Generate a GPG key (optional)

Duplicity encrypts your backups with GPG.

You may also need random number generator for GPG:

apt-get install rng-tools
nano /etc/default/rng-tools
HRNGDEVICE=/dev/urandom
service rng-tools start

If you don’t have the GPG key or you want to create a new one for the backup purposes:

gpg --gen-key
and go though the wizard. Check if it has been created and write down the GPG key ID:

gpg --list-keys

pub 2048R/ 2014-05-27

pub 2048R/44B2DB8E 2014-06-12
Key fingerprint = 9D7E 4D0F 7377 26A7 B5F7 0E27 99E5 B79D 44B2 DB8E
uid Jan Faix
sub 2048R/9413BE41 2014-06-12

Don’t forget to backup your GPG key (~/.gnupg/*.gpg) to safe location!

Step 7. Backup file system and transfer the archive to SFTP server (+optional MySQL backup included)

Install duplicity-backup script.

wget -O /usr/local/bin/duplicity-backup.sh https://raw.githubusercontent.com/zertrin/duplicity-backup/master/duplicity-backup.sh
wget -O /usr/local/etc/duplicity-backup.conf https://raw.githubusercontent.com/zertrin/duplicity-backup/master/duplicity-backup.conf.example

chmod 755 /usr/local/bin/duplicity-backup.sh
chmod 600 /usr/local/etc/duplicity-backup.conf

Check if /usr/local/bin is in PATH environment “echo $PATH”. If not, edit ~/.bash_profile and modify “PATH=$PATH:/usr/local/bin” entry.

Edit duplicity-backup script (/usr/local/bin/duplicity-backup.sh), change the path to configuration file:

CONFIG="/usr/local/etc/duplicity-backup.conf"

Now, edit duplicity-backup script configuration file (/usr/local/etc/duplicity-backup.conf) and fill configuration parameters. The inline documentation is self-explanatory; here are values to adapt:

#AWS_ACCESS_KEY_ID="<your AWS Access Key ID>"
#AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>"
ENCRYPTION='yes'
PASSPHRASE="foobar_gpg_passphrase"
GPG_ENC_KEY="foobar_gpg_key"
GPG_SIGN_KEY="foobar_gpg_key"
ROOT="/disk"
# ! Change server hostname !
DEST=ssh://faix@zviretnik.eu//mnt/data/.backups/home-server/
INCLIST=( "/disk/games" )
EXCLIST=( )
STATIC_OPTIONS="--full-if-older-than 7D"
STATIC_OPTIONS="--full-if-older-than 7D --s3-use-new-style --s3-european-buckets"
CLEAN_UP_TYPE="remove-older-than"
CLEAN_UP_VARIABLE="6M"
LOGDIR="/var/log/duplicity/"
LOG_FILE="duplicity-`date +%Y-%m-%d_%H-%M`.log"
LOG_FILE_OWNER="root:root"
VERBOSITY="-v4"
EMAIL_TO=root@faix.cz
EMAIL_SUBJECT="duplicity-backup, `hostname -s`, `date +%Y-%m-%d`"

Test the backup script:

duplicity-backup.sh --backup

ssh faix@zviretnik.eu "ls -la /mnt/data/.backups/home-server/"

You should get the a list similar to this one:

-rw-r--r-- 1 faix users    26255076 Jun 12 22:46 duplicity-full.20140612T204318Z.vol1.difftar.gpg
-rw-r--r-- 1 faix users     4849664 Jun 12 22:46 duplicity-full.20140612T204318Z.vol2.difftar.gpg

Check if you can restore from the backup (quite important issue :)):

mkdir /tmp/restore
duplicity-backup.sh --restore /tmp/restore

If everything works fine, orchestrate duplicity S3 backup from Backupninja. Create a new action (/etc/backup.d/30.backup_to_s3.sh) with following content (/usr/local/bin/duplicity-backup.sh --backup).

The numerical prefix must be higher than of the optional MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (e.g. /etc, /var…), is pushed to the S3 bucket.

Don’t forget to set the owner, group and permissions of new action script:

chown root:root /etc/backup.d/30.backup_to_s3.sh
chmod 700 /etc/backup.d/30.backup_to_s3.sh

Step 8. Test everything together

backupninja --now --debug

and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.