FreeNAS is an open source storage platform based on FreeBSD that supports file sharing across Windows, Apple, and Unix/Linux systems. rsync is an open source file copying utility for Linux/Unix systems. It’s famous for its “delta-transfer” algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. It offers a large number of options, including the ability to copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon; and, the ability to preserve symbolic links, hard links, file ownership, permissions, devices and times.
In this post I will describe how to configure FreeBSD, FreeNAS and rsync to securely backup files located on FreeNAS to a FreeBSD machine located at a remote location. All steps involved assume you have a running implementation of FreeBSD and FreeNAS. The software versions used in this post were as follows:
There are two ways for the FreeBSD machine to contact FreeNAS using rsync: using a remote-shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP. We’re going to use ssh as our transport method. We’ll also create a new user account in FreeNAS named “rsync,” which we will enable access to via ssh but limit its use solely for the purpose of backing up files using rsync.
Begin by logging into the FreeNAS web interface, navigate to Access->Users and Groups->Users and select the “+” icon to create a new user account. The following parameters will need to be configured:
Name – This is the login name of the new user. You’re welcome to use any user name you’d like here. For purposes of our example though let’s enter rsync.
Full Name – This field should match the Name field, so enter rsync again here.
Password – Our plans include generating and using ssh keys to authenticate the FreeBSD machine to the FreeNAS machine and not passwords, so leave these fields blank.
User ID – Unless you have a specific reason to change the User ID you should retain whatever FreeNAS has chosen for you here, typically UID 1001 if this is your first new user account on FreeNAS.
Shell – This parameter specifies the login shell for our new user rsync. The nologin option should not be selected as it will not permit access to the account. For purposes of our example, we’ll select the good old sh shell.
Primary group – This parameter sets the primary group that the user rsync will belong to. Unless you have a specific reason to change the group, the default group Guest should be retained here.
Additional group – No other group memberships need be selected.
Home directory – Here’s where we’ll enter the path to the home directory for the rsync user, which for purposes of our example should be set to /usr/home/rsync.
When complete, the configuration should resemble Figure 1. Now select “Add” to accept the changes and add rsync as a new user.
Now let’s configure and activate the rsync service in FreeNAS. Navigate to Services->Rsync->Server->Settings and check the “Enable” box, then click on the drop down list under “Map to user” and select rsync. The remaining fields can remain at their default settings. When complete, the configuration should resemble Figure 2. Now select “Save and Restart” to accept the changes and start the rsync server.
Next, we need to ensure the ssh service is running in FreeNAS. Navigate to Services->SSH and check the “Enable” box. To help improve security, we’ll avoid using the default TCP port 22 and, for purposes of our example, use port 13725 instead. Check the “Permit root login” box and ensure that “Password authentication” box is checked. When complete, the configuration should resemble Figure 3. Now select “Save and Restart” to accept the changes and start the ssh server.
Finally, connect via ssh to the the root account of the FreeNAS machine and create the directory /usr/home/rsync/.ssh, which we’ll use to hold the public RSA key that we’ll eventually generate and copy from the FreeBSD machine. Note: Alternatively, you can create this directory using FreeNAS’s built-in file management tools by navigating to Advanced->File Manager. Make sure to provide the correct local IP address or host name of the FreeNAS machine. If this machine is located on another network, then make sure to provide the correct IP address or URL of the network where the FreeNAS machine is located, as well as ensure that any NAT gateway is configured to pass TCP port 13725 traffic to the correct host:
ssh -p 13725 root@<freenas-ip-address> mkdir /usr/home/rsync/.ssh
Okay, we’re finished with thr FreeNAS configuration for now. Let’s move on and configure the FreeBSD machine.
The following steps assume you have the FreeBSD Ports Collection installed. If not, you can install it by using the following commands:
portsnap fetch portsnap extract
If the Ports Collection is already installed, make sure to update it:
portsnap fetch update
Navigate to the rsync port and build it, accepting the default configuration options:
cd /usr/ports/net/rsync make install clean
Next, we’ll choose a location that will be used to backup the FreeNAS files. For purposes of our example, we will create the subdirectory backup within the primary user’s home directory on the FreeBSD machine:
Then make sure the user owns this directory, as well as any files and subdirectories within it:
chown -R iceflatline /home/iceflatline/backup
Recall that we shunned conventional password authentication when creating the rsync user account on FreeNAS. Instead, we’re going to generate an RSA public/private key pair on the FreeBSD machine, and copy the public key to the FreeNAS machine. rsync will be able to login and authenticate itself to the FreeNAS machine without the need for a password because the FreeBSD machine has the corresponding private key. rsync can now run autonomously because it will not be prompted for a password each time it attempts to login:
ssh-keygen –q –t rsa
By default the RSA keys are 2048 bits. If you’re really paranoid you can increase this to 4096 bits if desired with the -b flag:
ssh-keygen –q –t rsa –b 4096
You’ll be asked by the ssh-keygen script where it should store the keys (~/.ssh is the default), and then asked to enter and confirm a password for the private key that will be stored in this machine. In our particular case, we don’t want to be prompted for a password so simply press enter. The private key (id_rsa) and the public key (id_rsa.pub) are generated and then stored in ~/.ssh (if you accepted the default directory). Now that our keys are generated, let’s copy the public key to the FreeNAS machine using the venerable secure copy command:
scp -P 13725 /home/iceflatline/.ssh/id_rsa.pub root@<freenas-ip-address>:~
Note that in the case of the scp command we must designate the ssh port number using the upper-case -P option, and pass to it the same ssh port number that was designated when we configured FreeNAS. Here again, make sure to provide the correct local IP address or host name of the FreeNAS machine or, if this machine is located on another network, the correct IP address or URL of the network where the FreeNAS machine is located. When you successfully connect you’ll likely receive a message concerning the authenticity of the FreeNAS machine, along with a fingerprint of its public RSA key, and asked if you’re sure you want to continue connecting. Accept by typing yes and you’ll be presented with a password prompt for the root user on the FreeNAS machine. Enter the password and the public key will be copied.
Now let’s connect again via ssh to the root account of the FreeNAS machine and copy the public key to /home/rsync/.ssh/authorized_keys so that rsync can securely connect to FreeNAS without the need for a password. We’ll also take this opportunity to remove the public key from the root account:
cat /root/id_rsa.pub >> /home/rsync/.ssh/authorized_keys rm /root/id_rsa.pub
Then lock down permissions and make sure the user rsync owns of all files and subdirectories within its home directory:
chmod 700 /home/rsync/.ssh chmod 600 /home/rsync/.ssh/authorized_keys chown -R rsync /home/rsync
Finally, let’s make sure we can connect to the rsync account on FreeNAS using only public/private key authentication:
ssh -p 13725 rsync@<freenas-ip-address>
When you successfully connect you’ll once again receive a message concerning the authenticity of the FreeNAS machine. Accept by typing yes and you’ll be securely connected to the rsync user’s home directory (/home/rsync/) on FreeNAS without the need to enter a password.
Okay, let’s move on and configure rsync.
For purposes of example, let’s assume that on the FreeNAS machines there is directory /mnt/files/foo containing a number of files along with two subdirectories – /mnt/files/foo/bar1 and /mnt/files/foo/bar2 – that we wish to backup on a routine basis to the FreeBSD machine. You could issue the following command to the FreeBSD machine accomplish that:
rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup
Long command right? Let’s walk through it. First, we invoke the rsync command and pass to it the following options:
-a – Specifies that the files are transferred in “archive” mode, which ensures that any symbolic links, devices, attributes, permissions, ownerships etc are preserved.
-v - Increases the verbosity of the command so that you know in a little more detail what rsync is/is not doing.
-z - Specifies that rsync should use compression to reduce the size of data portions of its file transfers.
–delete – Specifies that rsync should delete files in the destination that no longer exist on the source side. This helps to keep the file systems specified in the source and destination synchronized.
Then we tell rsync that we’d like to use ssh and port 13725 by using the “-e ssh -p 13725″ option. Finally, we tell rsync what the source and destination file systems are.
Another rsync option that I find handy is –exclude. Let’s say that for whatever reason you’d like to exclude the subdirectory /mnt/files/foo/bar2 and all the files therein. You’d use the following command to accomplish that:
rsync -avz --delete --exclude bar2/ "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup
Please consult the rsync man pages for further information on what rsync options are available for your specific needs.
Once you get the rsync command configured the way you’d like it, then it’s time to add it to cron on the FreeBSD machine so that it will perform backups on a routine basis. cron is a *nix utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon. These tasks are typically referred to as “cron jobs.” A “crontab” is a file which contains one or more cron job entries to be run at specified times. You can create a new crontab (or edit an exiting one) using the system’s default command line editor by using the command crontab -e under your user account. Here’s some example cron job entries using our rsync command. You could add one or many rsync cron job entries to your crontab depending on your needs, then simply uncomment the one you want to use. Also, cron will attempt to email to the user the output of the commands it runs. To silence this, we’ll redirect the command output to /dev/null:
#Backup every 15 minutes #*/15 * * * * /usr/local/bin/rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup >> /dev/null 2>&1 #Backup every 30 minutes #*/30 * * * * /usr/local/bin/rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup >> /dev/null 2>&1 #Backup at the top of every hour 0 * * * * /usr/local/bin/rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup >> /dev/null 2>&1 #Backup every three hours at the top of each hour #0 */3 * * * /usr/local/bin/rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup >> /dev/null 2>&1 #Backup every six hours at the top of each hour #0 */6 * * * /usr/local/bin/rsync -avz --delete "-e ssh -p 13725" rsync@<freenas-ip-address>:/mnt/files/foo/ /home/iceflatline/backup >> /dev/null 2>&1
While placing the full rsync command in the crontab works just fine, it can become a bit unwieldy, particularly if the command is lengthy. A more elegant approach is to capture the command and associated options in a shell script and invoke the script in a crontab instead of the full command. Here’s what a basic shell script might look like, again utilizing our example rsync command:
#!/bin/sh # A bit about RSYNC_ARGS: # -a=archive, -v=verbose, -z=compress with syncing # --delete means it will delete in LPATH anything not in RPATH # Adding the option -- exclude will exclude specified directories. # The trailing / specifies the entire directory and all files within. # For example, RSYNC_ARGS="-avz --delete --exclude bar2/" will # exclude the directory bar2 and all its files. ### START OF SCRIPT RSYNC=/usr/local/bin/rsync RSYNC_ARGS="-avz --delete --exclude bar2/" RSYNC_PORT="-e ssh -p 13725" RUSER=rsync RHOST="freenas-ip-address" RPATH=/mnt/files/foo/ LPATH=/home/iceflatline/backup DATE=/bin/date LOG=/home/iceflatline/cronlog # Let's create a blank line between # log entries so their not all bunched together echo "" >> $LOG # And precede each log entry with the current date/time $DATE >> $LOG # Now let's run the command and log it $RSYNC $RSYNC_ARGS "$RSYNC_PORT" $RUSER@$RHOST:$RPATH $LPATH >> $LOG 2>&1 # Finally, let's mark the end of our log entry with a delimiter echo "**********" >> $LOG ### END OF SCRIPT
You’ll notice that in addition to running the rsync command, the script will also create the log file /home/iceflatline/cronlog and append the date/time and a brief summary of what transpired to it every time script runs. To use this script, modify its values to suit your particular need, including the correct IP address or URL for the FreeNAS machine, and give it a name, say for example rsync-freenas. Then create a bin/ subdirectory in the home directory on the FreeBSD machine, copy the file to that location, and make it executable:
mkdir /home/iceflatline/bin chmod +x /home/iceflatline/bin/rsync-freenas
Now then, a crontab entry invoking this script would look like this; i.e., a whole lot less complex:
#Backup at the top of every hour #0 * * * * /home/iceflatline/bin/rsync-freenas
Now that we have everything configured, login again to the FreeNAS web interface. Navigate to Services->SSH and uncheck both the “Permit root login” and “Password authentication” boxes. When complete, select “Save and Restart” to accept the changes and restart the ssh server.
Here’s a few of tips that may help improve your experience when using rsync in the manner described in this post
This concludes the post on how to remotely and securely backup and synchronize files located on a FreeNAS machine to a FreeBSD-based machine, using rsync, a fast and versatile file copying tool.