Rolling Backup with rsync

Source

Author: akeil
Date: 2014-03-21
Version: 2
updated: 2014-03-24

This post describes a backup solution for a constellation where the source machine - the one that holds the data to be backed up - is "always running". The target machine - where backups are stored - is not always available.

More precisely we are backing up data from a RaspberryPi to a desktop computer.

The backup is controlled from the target machine, where it is scheduled to run periodically with anacron.

Most of this is taken from this website [1].

Rolling Backup

A rolling backup is used, where we create one backup per day and keep the n most recent backups.

Daily backups are kept for seven days. Every week, the oldest of the daily backups is kept as a weekly backup. Weekly backups are kept for 3 Weeks.

The resulting directory structure is this:

/backup-root
    /backup.daily.0
    /backup.daily.1
    /backup.daily.2
    /backup.daily.3
    /backup.daily.4
    /backup.daily.5
    /backup.daily.6
    /backup.weekly.0
    /backup.weekly.1
    /backup.weekly.2

RSync

rsync [2] is used to transfer data from the source to the target machine and to manage the backed up files. rsync also allows to create incremental backups by hard-linking files that were unchanged from older backups.

rsync can be used to sync data over the network but it can also work to sync directories athe same machine.

Install rsync and start the rsyncd service (Arch Linux):

# pacman -S rsync
# systemctl enable rsyncd
# systemctl start rsyncd

If a the iptables firewall is is used, allow the backup-machine to connect to rsyncd listening on port 873:

-A TCP -p tcp -m tcp --src 192.168.2.0/24 --dport 873 -j ACCEPT

Assumes that a chain named TCP is set up that handles TCP traffic. Also assumes that 192.168.2.* is your local network and that the basic policy is to DROP everything that is not explicitly allowed.

rsync will make only make directories available if they are configured in a module in /etc/rsyncd.conf. From the rsync documentation:

... each module exports a directory tree as a symbolic name. Modules are exported by specifying a module name in square brackets [module] followed by the parameters for that module. The module name cannot contain a slash or a closing square bracket. If the name contains whitespace, each internal sequence of whitespace will be changed into a single space, while leading or trailing whitespace will be discarded.
[home.username]
    path = /home/username
    comment = My Home
    read only = yes
    uid = username
    gid = username

You address it like so:

$ rsync rsync://HOST/MODULE

Backup-Script

Daily Backups

#!/bin/sh

# Backup script for radicale calendar data.
# Data is stored on a remote machine which is assumed to be always running.
# The source machine must run rsyncd.
# The backup is initialized from a client that is not always on.
# Intended to be run as a daily job via anacron.
# rsync is required on the target machine.

set -o errexit
unset PATH  # avoid accidental use of $PATH

# Source machine, hostname or IP
SRC_HOST=my-host
# rsync-module on host
SRC_MODULE=my-module
# target
BACKUP_ROOT=/mnt/storage/backup

# End Configuration ----------------------------------------------------------


# Define and create backup directories.
b0="$BACKUP_ROOT/backup.daily.0"
b1="$BACKUP_ROOT/backup.daily.1"
b2="$BACKUP_ROOT/backup.daily.2"
b3="$BACKUP_ROOT/backup.daily.3"
b4="$BACKUP_ROOT/backup.daily.4"
b5="$BACKUP_ROOT/backup.daily.5"
b6="$BACKUP_ROOT/backup.daily.6"

/bin/mkdir -p "$b0"
/bin/mkdir -p "$b1"
/bin/mkdir -p "$b2"
/bin/mkdir -p "$b3"
/bin/mkdir -p "$b4"
/bin/mkdir -p "$b5"
/bin/mkdir -p "$b6"

# discard oldest backup
if [ -d "$b6" ]; then
    /bin/rm -rf "$b6"
fi

# move remaining backups up by one level
if [ -d "$b5" ]; then
    /bin/mv "$b5" "$b6"
fi

if [ -d "$b4" ]; then
    /bin/mv "$b4" "$b5"
fi

if [ -d "$b3" ]; then
    /bin/mv "$b3" "$b4"
fi

if [ -d "$b2" ]; then
    /bin/mv "$b2" "$b3"
fi

if [ -d "$b1" ]; then
    /bin/mv "$b1" "$b2"
fi

if [ -d "$b0" ]; then
    /bin/mv "$b0" "$b1"
fi

# fetch a new backup
src="rsync://$SRC_HOST/$SRC_MODULE/"
/usr/bin/rsync --archive --delete --link-dest "../backup.1" "$src" "$b0"

# have mtime reflect backup date
/usr/bin/touch "$b0"

Rotate Backups

The script for rotating weekly backups:

#!/bin/bash

# Rotate Weekly Backups
# Drop the oldest weekly backup,
# copy the oldest daily backup to become the youngest weekly backup.
# Intended to be run weekly via anacron.

set -o errexit
unset PATH

BACKUP_ROOT=/mnt/storage/backup
OLDEST_DAILY_BACKUP="$BACKUP_ROOT/backup.daily.6"


# ----------------------------------------------------------------------------

b0="$BACKUP_ROOT/backup.weekly.0"
b1="$BACKUP_ROOT/backup.weekly.1"
b2="$BACKUP_ROOT/backup.weekly.2"

/bin/mkdir --parents "$b0"
/bin/mkdir --parents "$b1"
/bin/mkdir --parents "$b2"

# delete oldest backup
if [ -d "$b2" ]; then
    /bin/rm -rf "b2"
fi

# move previous backups up one level
if [ -d "$b1" ]; then
    /bin/mv "$b1" "$b2"
fi

if [ -d "$b0" ]; then
    /bin/mv "$b0" "$b1"
fi

# copy the latest daily backup to becom the youngest weekly backup
if [ -d "$OLDEST_DAILY_BACKUP" ]; then
    /bin/cp --archive --link "$oldest_daily_backup" "$b0"
fi
[1] http://www.mikerubel.org/computers/rsync_snapshots/
[2] https://rsync.samba.org/