Rolling Backup with rsync
Rolling Backup with rsync
- author
-
akeil
- date
-
2014-03-21
- version
-
2
- updated
-
2014-03-24
This post describes a backup solution for a constellation where the source machine - the one that holds the data to be backed up - is "always running". The target machine - where backups are stored - is not always available.
More precisely we are backing up data from a RaspberryPi to a desktop computer.
The backup is controlled from the target machine, where it is scheduled
to run periodically with anacron.
Most of this is taken from this website 1.
Rolling Backup
A rolling backup is used, where we create one backup per day and keep the n most recent backups.
Daily backups are kept for seven days. Every week, the oldest of the daily backups is kept as a weekly backup. Weekly backups are kept for 3 Weeks.
The resulting directory structure is this:
/backup-root
/backup.daily.0
/backup.daily.1
/backup.daily.2
/backup.daily.3
/backup.daily.4
/backup.daily.5
/backup.daily.6
/backup.weekly.0
/backup.weekly.1
/backup.weekly.2
RSync
rsync 2 is used to transfer data from the source to the target machine
and to manage the backed up files.
rsync also allows to create incremental backups by hard-linking
files that were unchanged from older backups.
rsync can be used to sync data over the network
but it can also work to sync directories athe same machine.
Install rsync and start the rsyncd service (Arch Linux):
# pacman -S rsync # systemctl enable rsyncd # systemctl start rsyncd
If a the iptables firewall is is used, allow the backup-machine
to connect to rsyncd listening on port 873:
-A TCP -p tcp -m tcp --src 192.168.2.0/24 --dport 873 -j ACCEPT
Assumes that a chain named TCP is set up that handles TCP traffic.
Also assumes that 192.168.2.* is your local network
and that the basic policy is to DROP everything that is not explicitly
allowed.
rsync will make only make directories available if they are configured
in a module in /etc/rsyncd.conf. From the rsync documentation:
... each module exports a directory tree as a symbolic name. Modules are exported by specifying a module name in square brackets [module] followed by the parameters for that module. The module name cannot contain a slash or a closing square bracket. If the name contains whitespace, each internal sequence of whitespace will be changed into a single space, while leading or trailing whitespace will be discarded.
[home.username] path = /home/username comment = My Home read only = yes uid = username gid = username
You address it like so:
$ rsync rsync://HOST/MODULE
Backup-Script
Daily Backups
#!/bin/sh # Backup script for radicale calendar data. # Data is stored on a remote machine which is assumed to be always running. # The source machine must run rsyncd. # The backup is initialized from a client that is not always on. # Intended to be run as a daily job via anacron. # rsync is required on the target machine. set -o errexit unset PATH # avoid accidental use of $PATH # Source machine, hostname or IP SRC_HOST=my-host # rsync-module on host SRC_MODULE=my-module # target BACKUP_ROOT=/mnt/storage/backup # End Configuration ---------------------------------------------------------- # Define and create backup directories. b0="$BACKUP_ROOT/backup.daily.0" b1="$BACKUP_ROOT/backup.daily.1" b2="$BACKUP_ROOT/backup.daily.2" b3="$BACKUP_ROOT/backup.daily.3" b4="$BACKUP_ROOT/backup.daily.4" b5="$BACKUP_ROOT/backup.daily.5" b6="$BACKUP_ROOT/backup.daily.6" /bin/mkdir -p "$b0" /bin/mkdir -p "$b1" /bin/mkdir -p "$b2" /bin/mkdir -p "$b3" /bin/mkdir -p "$b4" /bin/mkdir -p "$b5" /bin/mkdir -p "$b6" # discard oldest backup if [ -d "$b6" ]; then /bin/rm -rf "$b6" fi # move remaining backups up by one level if [ -d "$b5" ]; then /bin/mv "$b5" "$b6" fi if [ -d "$b4" ]; then /bin/mv "$b4" "$b5" fi if [ -d "$b3" ]; then /bin/mv "$b3" "$b4" fi if [ -d "$b2" ]; then /bin/mv "$b2" "$b3" fi if [ -d "$b1" ]; then /bin/mv "$b1" "$b2" fi if [ -d "$b0" ]; then /bin/mv "$b0" "$b1" fi # fetch a new backup src="rsync://$SRC_HOST/$SRC_MODULE/" /usr/bin/rsync --archive --delete --link-dest "../backup.1" "$src" "$b0" # have mtime reflect backup date /usr/bin/touch "$b0"
Rotate Backups
The script for rotating weekly backups:
#!/bin/bash # Rotate Weekly Backups # Drop the oldest weekly backup, # copy the oldest daily backup to become the youngest weekly backup. # Intended to be run weekly via anacron. set -o errexit unset PATH BACKUP_ROOT=/mnt/storage/backup OLDEST_DAILY_BACKUP="$BACKUP_ROOT/backup.daily.6" # ---------------------------------------------------------------------------- b0="$BACKUP_ROOT/backup.weekly.0" b1="$BACKUP_ROOT/backup.weekly.1" b2="$BACKUP_ROOT/backup.weekly.2" /bin/mkdir --parents "$b0" /bin/mkdir --parents "$b1" /bin/mkdir --parents "$b2" # delete oldest backup if [ -d "$b2" ]; then /bin/rm -rf "b2" fi # move previous backups up one level if [ -d "$b1" ]; then /bin/mv "$b1" "$b2" fi if [ -d "$b0" ]; then /bin/mv "$b0" "$b1" fi # copy the latest daily backup to becom the youngest weekly backup if [ -d "$OLDEST_DAILY_BACKUP" ]; then /bin/cp --archive --link "$oldest_daily_backup" "$b0" fi