UPDATE 2011/03/09 – I updated the code to backup to an NFS mount, and to include the “-h 0″ flag to skip all nodump flags. That was causing me serious problems.

Summary

I’ve given quite a bit of thought to backup procedures at home since my FreeBSD 8.1 box dropped my mirrored filesystem. The signs of impending apocalypse were there, I just didn’t pay them proper heed. Fortunately, all of my data was salvaged; unfortunately, I lost all the custom PHP code I wrote over the last 6 months, my wordpress themes, plugins and modifications, and everything else that actually DID anything with all that data. So, while I’ve been rewriting that, I’ve been giving equal, if not more attention to backing it up. I’ll catch up again, but before I do that, I’ll make sure I won’t fall behind again.

I did a few searches for FreeBSD backup solutions, and rolled my own little backup script using dump. It was decent, but it didn’t do everything I wanted as well as I wanted it to. Every night was a full backup, and there were no incrementals. I had to implement some pretty inelegant code to accomplish a couple things simply b/c I didn’t know how else to do it. So I kept looking and eventually zeroed in on David Andrzejewski’s work. He clearly states what he put out there is a use-at-your-own risk kind of script. I took it anyway as a starting block, and fleshed it out for my own purposes.

My requirements were similar to his, with the exception that I don’t have a cloud based storage account at the time of this writing and instead will be using a removable USB connected storage drive.

Project Goals

• Run with native or easily accessible tools.
• Full off-system backup of entire system once a week.
• Incremental off-system backup of entire system nightly.
• Separate off-system backups of individual critical files to make future restores easier.

Future Goals

• Play with ${DUMPCACHE} to see how it affects the time to execute in my environment. Drop it back to 8MB for a week. Ramp it up to 64MB for a week. Recommended is 32MB, but it’s a party! Let’s see what happens.
• Continue monitoring and fine tuning the hardware, OS environment and script to ensure maximum performance and stability. I haven’t recompiled a kernel in a while, maybe I’ll see about that.
• These are “as money allows” goals. I’m sure my wife is getting tired of me spending money on hardware. Then again, she does appreciate that I have a hobby that keeps me off the streets and out of the brothels.
  • Continue looking for consumer level, but sufficiently robust NAS solutions featuring RAID5 mirroring and access via secure and/or open protocols (ssh, smb, rsync, etc.) to replace (or augment) the removable drives I’m using now. No Windows-Only solutions please.
  • Evaluate cloud based storage for off-site backups. I’m looking at SpiderOak right now at the recommendation of a friend. I like their zero-knowledge solution and pricing, but more research is required. We’ll have upwards of 500GB of storage requirements, so we’ll have to weigh the monetary costs of cloud storage and bandwidth usage carefully against the risk of my solution failing when (!if) I need to restore. For the moment, I’m relatively comfortable with dumping the filesystems to removable drives, and keeping certain ultra-critical bits of recovery text (bsdlabels, fstabs, choice config files, etc.) in Google Docs.

My Environment

Two physically identical servers built from the ground up running FreeBSD 8.1. Each system houses a 150GB system drive (/dev/ad4s1) and a 500GB data/storage drive (/dev/ad6s1), and runs with 2GB of RAM.

I have /, /usr and /var mounted individually on the 150GB drive, and /home (containing /users and /www) mounted on the 500GB drive. I thought about getting separate drives for /www and /home, but decided I didn’t want to deal with planning for storage allocation. Instead I created /home/www for web files, and /home/users for user accounts. It’s not exactly standard, but it’s not unprecedented, and I make it work.

Off my “production” server I’ve hung a 2.5″ 320GB USB2.0 removable drive. Off my “development” server I’ve hung a 2.5″ 100GB USB2.0 removable drive. I’ll adjust the size of the drives as needed. That’s just what I had on hand. Both were UFS formatted using fdisk.

During the backup job, those drives are mounted at /backup. The rest of the time they’re plugged in, but not mounted.

The Script

You’re welcome to this, but be warned, if it borks up your machine, destroys your pr0n collection, or sends terrifying space monkeys into your engine room, don’t blame me. Use at your own risk. There, now that I’m all disclaimed…

#!/usr/local/bin/bash

# Much appreciation to David Andrzejewski, and the work he started at
# http://www.davidandrzejewski.com/2010/03/01/freebsd-backup-using-dump-and-duplicity/
# I'm sure his current script/processes far outstrip this, but this
# is my (f)stab at it

# Version: 0.5
# * Provides 1 set of full backups and 6 associated incrementals
# * Backup files stored on mounted USB drive only

# I would like to see...
# * Writing to NAS with RAID5 and standard access (ssh, SMB, etc.)
# * Retrieve the cloud based storage interaction I stripped out

# DUMPLVL: provided via a command line flag ${1})
# WEEKDAY: provided via a command line flag ${2})

# HOSTNAME: The host being backed up. Used in informational messages
HOSTNAME=$( hostname )

# FSLIST: The list of file systems that will be dumped along with the
# name of the dump Example: /dev/ad4s1a=root will dump the /dev/ad4s1a
# volume and name it DDD.root.dump.levelN.bz2 where "N" is the dump level
# and "DDD" is the weekday
FSLIST="/dev/ad4s1a=root /dev/ad4s1d=var /dev/ad4s1f=usr /dev/ad6s1d=home"

# BSDLABEL_PARTITIONS: The list of partitions to run `bsdlabel` on
# This will be saved in the backup directory during runtime as
# ${WEEKDAY}.bsdlabel_${PARTITION}.txt
BSDLABEL_PARTITIONS="ad4s1 ad6s1"

# DUMPDEVICE: The location the files will be dumped to
DUMPDEVICE=sosaria:/home/dumps/${HOSTNAME}

# DUMPDIR: The directory that the dumps will be written to
DUMPDIR=/backup

# STAGINGDIR: The directory where dumps are stored before being written
# to ${DUMPDIR}
STAGINGDIR=/home/dumps/stage

# ARCHIVEDIR: The local directory dumps are stored after being written
# to ${DUMPDIR}
ARCHIVEDIR=/home/dumps/${HOSTNAME}

# NODUMP_DIRS: List of directories to set the nodump flag
NODUMP_DIRS="/usr/ports /usr/obj /usr/src /home/www/logs /home/www/src /home/dumps"

# DUMPCACHE: The amount of memory to give dump
DUMPCACHE=32

# DUMPFLAGS: The flags to feed dump
DUMPFLAGS="uanL -h 0 -f"

# FSTYPE: The filesystem type of the mounted partition
FSTYPE=nfs

# These should be standard

# BSDLABELCMD: The bsdlabel command
BSDLABELCMD=/sbin/bsdlabel

# DUMPCMD: The dump command
DUMPCMD=/sbin/dump

# MOUNTCMD: The mount command
MOUNTCMD=/sbin/mount

# UMOUNTCMD: The mount command
UMOUNTCMD=/sbin/umount

##---------------------------------------------------------------------
# Shouldn't have to edit anything below here

# Get the start time so we can gauge how long this is taking. Useful in
# tweaking ${DUMPCACHE}
START=$( date +%s )

# Get the directory we're running from
SCRIPTDIR=$( dirname $0 )

cd ${SCRIPTDIR}
if [ $? -ne 0 ]; then
       echo "ERROR: Unable to cd to ${SCRIPTDIR}! Aborting!"
       exit 1
fi

# If we were executed like "./whatever.sh" - set SCRIPTDIR to the pwd
if [ "${SCRIPTDIR}" == "." ]; then
       SCRIPTDIR=$( pwd )
fi

echo "Script is running from ${SCRIPTDIR}"

# Check the command line to make sure we have what we need from it
# First check for the dump level
if [ "${1}" == "" ]; then
       echo "Must specify dump level. Aborting!"
       exit
else
       DUMPLVL=${1}
fi

# Sanity check
if [ "${DUMPLVL}" == "" ]; then
       echo "ERROR: For some reason DUMPLVL never got set! Aborting!"
       exit 1
fi

# Then get the weekday name off the command line
if [ "${2}" == "" ]; then
       echo "Must specify weekday name. Aborting!"
       exit
else
       WEEKDAY=${2}
fi

# Sanity check
if [ "${WEEKDAY}" == "" ]; then
       echo "ERROR: For some reason WEEKDAY never got set! Aborting!!"
       exit 1
fi

# Create the flag file so we can't run more than one instance
if [ -f "${SCRIPTDIR}/myself.flg" ]; then
       echo "Script running?! ${SCRIPTDIR}/myself.flg exists! Aborting!"
       exit 1
else
       echo "Touching myself at ${SCRIPTDIR}/myself.flg"
       touch ${SCRIPTDIR}/myself.flg
fi

# Check for the existance of ${STAGINGDIR}
if [ ! -d "${STAGINGDIR}" ]; then
       mkdir ${STAGINGDIR}
       if [ $? = 1 ]; then
               echo "Could not create ${STAGINGDIR}!  Aborting!"
               echo "Removing ${SCRIPTDIR}/myself.flg"
               rm -f ${SCRIPTDIR}/myself.flg
               exit 1
       fi
fi

# Check for the existance of ${ARCHIVEDIR}
if [ ! -d "${ARCHIVEDIR}" ]; then
       mkdir ${ARCHIVEDIR}
       if [ $? = 1 ]; then
               echo "Could not create ${ARCHIVEDIR}!  Aborting!"
               echo "Removing ${SCRIPTDIR}/myself.flg"
               rm -f ${SCRIPTDIR}/myself.flg
               exit 1
       fi
fi

echo ""
for DIR in ${NODUMP_DIRS}; do
       echo "Setting nodump on ${DIR}"
       chflags -R nodump ${DIR}
done

echo ""
echo "Dump Level: ${DUMPLVL}"

# Preserve a copy of root's crontab (/root/crontab is
# manually created with `crontab -l > ~/crontab` with every change
echo ""
echo "Copying /root/crontab to ${STAGINGDIR}/${WEEKDAY}.root_crontab"
cp -f /root/crontab ${STAGINGDIR}/${WEEKDAY}.root_crontab

# Preserve a copy of fstab
echo "Copying fstab to ${STAGINGDIR}/${WEEKDAY}.fstab.txt"
cp -f /etc/fstab ${STAGINGDIR}/${WEEKDAY}.fstab.txt

# Preserve a week's worth of bsdlabel copies for each partition
for PARTITION in ${BSDLABEL_PARTITIONS}; do
       echo "Writing bsdlabel for ${PARTITION} -> ${STAGINGDIR}/${WEEKDAY}.bsdlabel_${PARTITION}.txt"
       ${BSDLABELCMD} ${PARTITION} > ${STAGINGDIR}/${WEEKDAY}.bsdlabel_${PARTITION}.txt
done

# Dump the filesystems!
for FSITEM in ${FSLIST}; do
       # Get the devicename
       FS=$( echo ${FSITEM} | awk -F= '{ print $1 }' )
       # Get the filesystem name
       NAME=$( echo ${FSITEM} | awk -F= '{ print $2 }' )
       DUMPFILE=${WEEKDAY}.${NAME}.level${DUMPLVL}.dump
       echo ""
       echo "Dumping ${FS} to ${STAGINGDIR}/${DUMPFILE} at dump level ${DUMPLVL}"
       echo ""
       echo "${DUMPCMD} -C${DUMPCACHE} -${DUMPLVL}${DUMPFLAGS} ${STAGINGDIR}/${DUMPFILE} ${FS}"
       ${DUMPCMD} -C${DUMPCACHE} -${DUMPLVL}${DUMPFLAGS} ${STAGINGDIR}/${DUMPFILE} ${FS}
done

# Test for an existing backup device mount and either use the existing
# mountpoint or mount our backup directory

MOUNTRESULTS=$( ${MOUNTCMD} | grep "${DUMPDEVICE} on ${DUMPDIR}" )

if [ "${MOUNTRESULTS}" == "" ]; then
       echo ""
       echo "Mounting ${DUMPDEVICE} on ${DUMPDIR}"
       ${MOUNTCMD} -t ${FSTYPE} ${DUMPDEVICE} ${DUMPDIR}
       if [ $? = 1 ]; then
               echo "  ... failed. Aborting!"
               echo "Removing ${SCRIPTDIR}/myself.flg"
               rm -f ${SCRIPTDIR}/myself.flg
               exit 1
       else
               echo "  ... succeeded"
       fi
else
       echo "${HOSTNAME}:${DUMPDEVICE} already mounted on ${DUMPDIR}"
fi

# Copy the files to ${DUMPDIR} and archive them to {$ARCHIVEDIR}
cd ${STAGINGDIR}
echo ""
for FILE in *; do
       echo "Copying ${FILE} to ${DUMPDIR}"
       cp ${FILE} ${DUMPDIR}/${FILE}
       if [ $? = 1 ]; then
               echo "... Failed to copy ${FILE}! You might want to see to that."
       else
               echo "Moving ${FILE} to ${ARCHIVEDIR}"
               mv ${FILE} ${ARCHIVEDIR}/${FILE}
       fi
done

# Get a snapshot of how the dump directory looks for verification
echo ""
echo "Recent Additions to ${DUMPDIR}:"
echo ""
ls -lt ${DUMPDIR} | tail -n +2 | head -n 8

# Umount the backup filesystem
echo ""
echo "Unmounting ${DUMPDIR}"
${UMOUNTCMD} ${DUMPDIR}
if [ $? = 1 ]; then
       echo "  ... failed. You might want to see to that."
else
       echo "  ... succeeded"
fi

# Clear the running flag
echo ""
echo "Removing ${SCRIPTDIR}/myself.flg"
rm -f ${SCRIPTDIR}/myself.flg
if [ -f "${SCRIPTDIR}/myself.flg" ]; then
       echo "  ... failed. You might want to see to that."
else
       echo "  ... succeeded"
fi

echo ""
echo "Backup of ${HOSTNAME} Complete"

END=$( date +%s )
RUNTIME=$(( ${END} - ${START} ))
H=$(( ${RUNTIME}/3600 ))
M=$(( ( ${RUNTIME}/60 ) % 60 ))
S=$(( ${RUNTIME} % 60 ))

echo "It took ${H} hrs, ${M} mins and ${S} secs with -C${DUMPCACHE} (${RUNTIME} secs)"
exit 0

The Crontab

Here’s how I’ve set up my crontab. Like Mr. Andrzejewski, I opted to keep the specifics regarding the type of backup and the day it’s run in cron, rather than build it into the script. While it does make for a slightly longer crontab, it simplifies the logic in the script considerably. At the end of the day, I just feel better about telling the script what kind of backup to run (full or incremental), and the weekday name to embed in the resulting filenames, rather than letting it determine it itself. It’s a control thing.

# Daily Backups of filesystems
# Full backups on Sunday. Incremental backups every other day.
30 0 * * 0 /root/bin/backup/backup_script.sh 0 Sun 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 1 /root/bin/backup/backup_script.sh 1 Mon 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 2 /root/bin/backup/backup_script.sh 1 Tue 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 3 /root/bin/backup/backup_script.sh 1 Wed 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 4 /root/bin/backup/backup_script.sh 1 Thu 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 5 /root/bin/backup/backup_script.sh 1 Fri 2>&1 /dev/null | mail -s "System Backup" dvicci
30 0 * * 6 /root/bin/backup/backup_script.sh 1 Sat 2>&1 /dev/null | mail -s "System Backup" dvicci

This will finally result in a list of files looking something like this come Sunday morning. Sort to taste.

backup/Sat.usr.level1.dump
backup/Sat.var.level1.dump
backup/Sat.root.level1.dump
backup/Sat.fstab.txt
backup/Sat.bsdlabel_ad6s1.txt
backup/Sat.bsdlabel_ad4s1.txt
backup/Sat.root_crontab.txt
backup/Fri.home.level1.dump
backup/Fri.usr.level1.dump
backup/Fri.var.level1.dump
backup/Fri.root.level1.dump
backup/Fri.fstab.txt
backup/Fri.bsdlabel_ad6s1.txt
backup/Fri.bsdlabel_ad4s1.txt
backup/Fri.root_crontab.txt
backup/Thu.home.level1.dump
backup/Thu.usr.level1.dump
backup/Thu.var.level1.dump
backup/Thu.root.level1.dump
backup/Thu.fstab.txt
backup/Thu.bsdlabel_ad6s1.txt
backup/Thu.bsdlabel_ad4s1.txt
backup/Thu.root_crontab.txt
backup/Wed.home.level1.dump
backup/Wed.usr.level1.dump
backup/Wed.var.level1.dump
backup/Wed.root.level1.dump
backup/Wed.fstab.txt
backup/Wed.bsdlabel_ad6s1.txt
backup/Wed.bsdlabel_ad4s1.txt
backup/Wed.root_crontab.txt
backup/Tue.home.level1.dump
backup/Tue.usr.level1.dump
backup/Tue.var.level1.dump
backup/Tue.root.level1.dump
backup/Tue.fstab.txt
backup/Tue.bsdlabel_ad6s1.txt
backup/Tue.bsdlabel_ad4s1.txt
backup/Tue.root_crontab.txt
backup/Mon.home.level1.dump
backup/Mon.usr.level1.dump
backup/Mon.var.level1.dump
backup/Mon.root.level1.dump
backup/Mon.fstab.txt
backup/Mon.bsdlabel_ad6s1.txt
backup/Mon.bsdlabel_ad4s1.txt
backup/Mon.root_crontab.txt
backup/Sun.home.level0.dump
backup/Sun.usr.level0.dump
backup/Sun.var.level0.dump
backup/Sun.root.level1.dump
backup/Sun.fstab.txt
backup/Sun.bsdlabel_ad6s1.txt
backup/Sun.bsdlabel_ad4s1.txt
backup/Sun.root_crontab.txt