Tag Archives: backup

Backup configuration on every Debian installation

The daily cron job does little bit more, actually.

  • create gzipped tar with /etc and other named directories
  • it creates list of extra installed packages from repositories
  • copy backup over LAN or Internet, checks if NFS disk is mounted or remote directory exists
  • it uses tar, cp and rsync commands

You can add it to cron via this commands:

touch /etc/cron.daily/backup-configuration
chown root:root /etc/cron.daily/backup-configuration
chmod 755 /etc/cron.daily/backup-configuration

/etc/cron.daily/backup-configuration

#!/bin/bash

## global variables

# change variables according to your needs

# destination remote directory to save backups
remote_directory=/disk/backup/home-desktop
# destination local NFS mount point
local_mount_point=/disk
# source local directory to store backups
local_directory=/var/backups
# destination SSH login and server name to save backups
remote_server_login=jan@faix.cz
# list of configuration directories and files you want to backup
backup_list="/etc \
             /usr/local/bin \
             /usr/local/etc \
             /var/spool/cron \
             /var/backups/*.0 \
             /var/backups/*.bak
             /var/backups/*.txt"
# parameter the script was launched with or set default LAN
if [ -z "$1" ]; then
    parameter=-LAN
  else
    parameter="$1"
fi

# do not change following variables
# local server name
machine_name=`hostname -s`

## functions
bck_cfg() {
  aptitude -F %p search "?and(?installed,?not(?automatic))" > $local_directory/pkglist.txt
  tar -czPf $local_directory/$machine_name.tar.gz $backup_list
}
start_job() { printf "\nStarting configuration backup job at `date +%c` ...\n"
}
job_complete() { printf "\nConfiguration backup job successfuly completed on host `hostname` at `date +%c`.\n"
}
dir_not() { printf "\nConfiguration backup job failed on host `hostname` at `date +%c`, $dir_loc does not exist.\n"
}

# check if local_directory does not exist, print error message and exit
if [ ! -d $local_directory ]; then
  dir_loc="local directory $local_directory"
  dir_not && exit
fi

# in case the parameter -LAN or -Internet is used, start configuration backup
case $parameter in
    -LAN)
    ## configuration backup job inside LAN
    dir_loc="local directory $local_directory"
    if [ ! -d $remote_directory ]; then
        mount $local_mount_point
    fi

    if [ -d $remote_directory ]; then
        start_job && bck_cfg
        cp $local_directory/$machine_name.tar.gz $remote_directory
        job_complete
      else
        dir_not
    fi
    ;;
    -Internet)
    ## configuration backup job over Internet
    dir_loc="remote directory $remote_directory"
    if (ssh $remote_server_login "[ ! -d $remote_directory ]"); then
        dir_not_ && exit
    fi

    if [ -d $local_directory ]; then
        start_job && bck_cfg
        rsync -azs --stats $local_directory/$machine_name.tar.gz $remote_server_login:$remote_directory/
        job_complete
      else
        dir_not
    fi
    ;;
    *)
    ## print help
    printf "\nThis script create machine configuration backup and copy it over network to destination server.\n"
    printf "\nConfiguration backup script must be run with one of the floowing parameter:\n"
    printf "\n\t-LAN\t\tbackup configuration within LAN to mounted NFS disk"
    printf "\n\t-Internet\tbackup configuration over Internet via rsync to destination folder\n"
    printf "\t\t\tDon't forget to exchange SSH keys running \"ssh-copy-id $remote_server_login\" from source machine you launch backup configuration script.\n"
    printf "\nCheck variables defined in begining of this script:\n"
    printf "\n\tremote_directory=$remote_directory"
    printf "\n\tlocal_mount_point=$local_mount_point"
    printf "\n\tlocal_directory=$local_directory"
    printf "\n\tremote_server_login=$remote_server_login"
    printf "\n\tbackup_list=$backup_list"
    printf "\n\tmachine_name=$machine_name"
    printf "\n\tdefault parameter=$parameter"
    printf "\n\n"
    ;;
esac

Backup data to remote SFTP server

http://grysz.com/2011/04/04/configure-amazon-s3-backup-with-backupninja-and-duplicity/

Data backup plan:

* pick desired directories with your data you want to backup
* you may backup system directories of your choice (in principle /etc /root /home /usr/local /srv etc.)
* optionally you may backup MySQL database

Used tools (tested on Debian Jessie and Ubuntu 12.04):

* BackupNinja 0.9.10 – for MySQL database backup and backup orchestration
* Duplicity 0.6.23 – filesystem backup
* duplicity-backup – wrapper shell script performing a backup using duplicity

Step 1. Install the tools

apt-get install backupninja duplicity python-paramiko python-lockfile dialog

! Do NOT install Duplicity 0.6.18 from Ubuntu 12.04 default repository, this version has buggy ssh client paramiko library !

Step 2. Configure Backupninja schedule

Check Backupninja configuration (/etc/backupninja.conf) if it suits your needs. The most important parameter is when it controls the backup schedule. I left it with the default value (everyday at 01:00). If you want to get backup report, fill at least `reportemail` parameter value.

Backupninja run every hour from cron job /etc/cron.d/backupninja to launch scheduled backup tasks.

Step 3. Backup MySQL database (optional)

Create a Backupninja backup action for MySQL database dump. The easiest way is to start “ninjahelper” tool and go through create a new backup action/mysql database backup wizard:

Directory to store backups: /var/backups/mysql
Backup all of databases: Yes
MySQL authentication method: password
Create a backup using mysqldump: sqldump
Compress the SQL output files: compress

Ninjahelper creates the action configuration in the /etc/backup.d/20.mysql file.

Perform a test:

backupninja --now --debug

Check if there has been no error messages and if the dump has been created in /var/backups/mysql/sqldump.

Step 4. Backup list of installed packages

echo "aptitude -F %p search \"?and(?installed,?not(?automatic))\" > /var/backups/pkglist.txt" > /etc/backup.d/25.pkglist.sh

The numerical prefix must be lower than last backup action.

Set the owner, group and permissions of new action script:

chown root:root /etc/backup.d/25.pkglist.sh
chmod 700 /etc/backup.d/25.pkglist.sh

Step 5. Exchange SSH key with backup server

From the server, which should be backed up run following commands as `root` user.

* Check if SSH keys already exists: ls ~/.ssh/id_rsa*
* If it doesn’t exist yet, generate them running command: ssh-keygen
* Install root user public key to backup server: ssh-copy-id ue]tod[kinterivznull]ta[xiaf
* Add known host key to known_hosts running: ssh-keyscan -t rsa zviretnik.eu>>~/.ssh/known_hosts

Step 6. Generate a GPG key (optional)

Duplicity encrypts your backups with GPG.

You may also need random number generator for GPG:

apt-get install rng-tools
nano /etc/default/rng-tools
HRNGDEVICE=/dev/urandom
service rng-tools start

If you don’t have the GPG key or you want to create a new one for the backup purposes:

gpg --gen-key
and go though the wizard. Check if it has been created and write down the GPG key ID:

gpg --list-keys

pub 2048R/ 2014-05-27

pub 2048R/44B2DB8E 2014-06-12
Key fingerprint = 9D7E 4D0F 7377 26A7 B5F7 0E27 99E5 B79D 44B2 DB8E
uid Jan Faix
sub 2048R/9413BE41 2014-06-12

Don’t forget to backup your GPG key (~/.gnupg/*.gpg) to safe location!

Step 7. Backup file system and transfer the archive to SFTP server (+optional MySQL backup included)

Install duplicity-backup script.

wget -O /usr/local/bin/duplicity-backup.sh https://raw.githubusercontent.com/zertrin/duplicity-backup/master/duplicity-backup.sh
wget -O /usr/local/etc/duplicity-backup.conf https://raw.githubusercontent.com/zertrin/duplicity-backup/master/duplicity-backup.conf.example

chmod 755 /usr/local/bin/duplicity-backup.sh
chmod 600 /usr/local/etc/duplicity-backup.conf

Check if /usr/local/bin is in PATH environment “echo $PATH”. If not, edit ~/.bash_profile and modify “PATH=$PATH:/usr/local/bin” entry.

Edit duplicity-backup script (/usr/local/bin/duplicity-backup.sh), change the path to configuration file:

CONFIG="/usr/local/etc/duplicity-backup.conf"

Now, edit duplicity-backup script configuration file (/usr/local/etc/duplicity-backup.conf) and fill configuration parameters. The inline documentation is self-explanatory; here are values to adapt:

#AWS_ACCESS_KEY_ID="<your AWS Access Key ID>"
#AWS_SECRET_ACCESS_KEY="<your AWS Secret Access Key>"
ENCRYPTION='yes'
PASSPHRASE="foobar_gpg_passphrase"
GPG_ENC_KEY="foobar_gpg_key"
GPG_SIGN_KEY="foobar_gpg_key"
ROOT="/disk"
# ! Change server hostname !
DEST=ssh://faix@zviretnik.eu//mnt/data/.backups/home-server/
INCLIST=( "/disk/games" )
EXCLIST=( )
STATIC_OPTIONS="--full-if-older-than 7D"
STATIC_OPTIONS="--full-if-older-than 7D --s3-use-new-style --s3-european-buckets"
CLEAN_UP_TYPE="remove-older-than"
CLEAN_UP_VARIABLE="6M"
LOGDIR="/var/log/duplicity/"
LOG_FILE="duplicity-`date +%Y-%m-%d_%H-%M`.log"
LOG_FILE_OWNER="root:root"
VERBOSITY="-v4"
EMAIL_TO=root@faix.cz
EMAIL_SUBJECT="duplicity-backup, `hostname -s`, `date +%Y-%m-%d`"

Test the backup script:

duplicity-backup.sh --backup

ssh faix@zviretnik.eu "ls -la /mnt/data/.backups/home-server/"

You should get the a list similar to this one:

-rw-r--r-- 1 faix users    26255076 Jun 12 22:46 duplicity-full.20140612T204318Z.vol1.difftar.gpg
-rw-r--r-- 1 faix users     4849664 Jun 12 22:46 duplicity-full.20140612T204318Z.vol2.difftar.gpg

Check if you can restore from the backup (quite important issue :)):

mkdir /tmp/restore
duplicity-backup.sh --restore /tmp/restore

If everything works fine, orchestrate duplicity S3 backup from Backupninja. Create a new action (/etc/backup.d/30.backup_to_s3.sh) with following content (/usr/local/bin/duplicity-backup.sh --backup).

The numerical prefix must be higher than of the optional MySQL backup action. In this configuration, first a MySQL dump is created and then, together with other directories (e.g. /etc, /var…), is pushed to the S3 bucket.

Don’t forget to set the owner, group and permissions of new action script:

chown root:root /etc/backup.d/30.backup_to_s3.sh
chmod 700 /etc/backup.d/30.backup_to_s3.sh

Step 8. Test everything together

backupninja --now --debug

and check if there has been no error messages, the backup is in the S3 bucket and it can be restored.

Aptitude

Záloha a obnova nainstalovaných balíčků

Pokud budete zálohovat Debian nebo na něm založenou distribuci, budete asi chtít zálohovat spíše seznam explicitně nainstalovaných balíčků než seznam všech nainstalovaných balíčků.

Pokud používáte nástroj Aptitude, pak víte, že tento nástroj si zapamatuje, co bylo nainstalováno jen jako závislost, a pokud pak odinstalujete původní balíček, Aptitude vám nabídne k odstranění i ty balíčky, které byly nainstalovány jako závislosti. To je velice užitečná funkce z hlediska údržby systému. Pokud byste zálohovali seznam všech balíčků, zahrnuli byste do toho i ony závislosti a po obnově systému (nebo při jeho migraci jinam) by váš Aptitude tuto svou funkci ztratil. Naštěstí není problém nechat si vypsat explicitně nainstalované balíčky:

aptitude -F %p search "?and(?installed,?not(?automatic))" > pkglist.txt

A posléze je nainstalovat do nového systému:

xargs aptitude --schedule-only install < pkglist.txt && aptitude install

Upgrade na nový release Ubuntu se provede příkazem: do-release-upgrade

Instalace klíče k novému repozitáři

V záhlaví PPA’s stránky najdete podpisový klíč “Signing key”. Bude vypadat přibližně takto:
1024R/72D340A3

Zkopírujte část následující za lomítkem, např.: 72D340A3

Nyní je třeba přidat klíč do vašeho systému, aby Ubuntu mohlo ověřit balíčky z PPA repozitáře.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 72D340A3

Nastavení automatických aktualizací pomocí anacron

apt-get install unattended-upgrades
/etc/apt/apt.conf.d/10periodic

APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "5";
APT::Periodic::Unattended-Upgrade "1";
APT::Periodic::RandomSleep "1800";

/etc/apt/apt.conf.d/50unattended-upgrades

Ubuntu

// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
// ${distro_id} and ${distro_codename} will be automatically expanded
"${distro_id} stable";
"${distro_id} ${distro_codename}-security";
"${distro_id} ${distro_codename}-updates";
};

Debian

Unattended-Upgrade::Origins-Pattern {
"o=Debian,a=testing";
"o=Debian,a=testing-updates";
"o=Debian,a=proposed-updates";
"origin=Debian,archive=${distro_codename},label=Debian-Security";
};

Unattended-Upgrade::Package-Blacklist {};
Unattended-Upgrade::Mail "root";
Unattended-Upgrade::MailOnlyOnError "true";

Test you configuration

unattended-upgrades --dry-run
unattended-upgrades --debug

and check the log:
cat /var/log/unattended-upgrades/unattended-upgrades.log