bookmark_borderHow to use Clonezilla to Create and Backup Disk Images to FreeNAS

FreeNAS is an open source storage platform based on FreeBSD that supports file sharing across Windows, Apple, and Unix/Linux systems. Clonezilla is an open source clone utility for doing bare-metal backup and recovery for disks and disk partitions.

In this post I will describe how to use Clonezilla to create image files of disks or disk partitions and backup those images in real time over a network to a machine running FreeNAS. All steps involved assume you have a running implementation of FreeNAS. The software versions used in this post were as follows:

  • Clonezilla v1.2.12-10
  • FreeNAS v0.7.1 Shere (revision 5127)

Configure FreeNAS

Let’s start by making a new directory in FreeNAS to hold the images we create using Clonezilla. For the purpose of example, let’s assume there is an existing mount point located at /mnt/files and create a new subdirectory called images at that location. Next, let’s enable Network File System (“NFS”) service in FreeNAS so that /mnt/files/images can be accessed from Clonezilla. Login into the FreeNAS web interface and navigate to Services -> NFS -> Settings and make sure that the check box for enabling NFS is checked then specify the number of servers that will run (Hint: the default value of four should easily handle dozens of users). Now select “Save and Restart.” Next, navigate to Services -> NFS -> Shares and select the “+” icon, where you are presented with the configuration screen for creating a new NFS share. Enter the path to be shared; the network that is authorized to access /mnt/files/images; and, make sure that the “All dirs” and “Quiet” check boxes are selected. The remaining options can remain at their defaults (See Figure 1). Then select “Add” then “Apply changes” (Note: If you would prefer to use Common Internet File System (“CIFS”) instead of NFS, navigate to Services -> CIFS/SMB and select the appropriate settings and add your network share).

Screenshot of NFS shared path configuration in FreeNAS

Figure 1

Using Clonezilla
Now it’s time to use Clonezilla. Download a copy of Clonezilla live and burn it to a CD (or place it on a bootable USB drive). Boot the system using the Clonezilla live disk, where you’ll be given the option to change the language and select an alternative keyboard keymap, eventually arriving at the “Start Clonezilla” Screen (See Figure 2).

Screenshot of Clonezilla's start screen

Figure 2

Highlight “Start Clonezilla” and select “Ok.” The screen that follows provides a choice of cloning/restoring a disk or disk partition using using an image, or cloning/restoring a disk or disk partition directly to another disk or disk partition. Since we’ll be creating an image of a disk, we should select “device-image work with disks or partitions using images” (See Figure 3).

Screenshot of Clonezilla menu option to work with with images or devices when cloning or restoring

Figure 3

Next, Clonezilla will ask us to define where the image will be saved to (or read from), providing you with several options for accomplishing this task. We’re interested in using an NFS server, so select “nfs_server Use NFS server” (See Figure 4).

Screenshot of Clonezilla options specifying where it should transfer images to/from

Figure 4

You’ll then be offered several choices on how Clonezilla should obtain IP network settings, including DHCP, static IP, PPPoE, or a shell prompt to manually define how it should go about doing obtain its settings. Select which option works best for you then select “Ok” (See Figure 5).

Screenshot of Clonezilla IP network configuration options

Figure 5

Clonezilla will then ask you whether it should use NFS version 2,3 or 4. Select “nfs NFS V2, V3,” then enter the IP address or the host name of your FreeNAS machine in the screen that follows. Next, you’re asked where the Clonezilla image will be saved to/read from. For the purpose of our example, we’ll enter “/mnt/files/images” to match the new directory we created earlier (See Figure 6).

Screenshot of Clonezilla option specifying where the image should be stored

Figure 6

Clonezilla will then mount /mnt/files/images and display its current disk usage. Select “Enter” to continue and you’ll be given a choice to use Clonezilla in either a beginner or expert mode. Select “Beginner mode: Accept the default options” and you’ll be presented with a choice of whether to create images of the entire disk and/or one or more partitions (Note: this screen is also where you would choose to restore disk or partition images). If you would prefer to create an image of a partition, select “saveparts Save_local_partitions_as_an_image” where you will be presented with a choice of which partition(s) to create images of. For the purpose our example, however, we’ll select “savedisk Save_local_disk_as_an_image” to create an image of the entire disk (See Figure 7).

Screenshot of Clonezilla options allowing user to specify whether to create or restore the images of disks or disk partitions

Figure 7

Next, we’ll enter a name for new image and then chose whether we want to check and repair the file system before Clonezilla creates the image. This option is only for supported for certain Linux files systems, like ext3/4, and not for NTFS, so unless you have one of the supported file systems and have reason to suspect the file system may be in error, you should simply select “Skip checking/repairing source file system.” Clonezilla will then ask if you’d like for it to check if the image it creates will be restorable. Don’t worry, it performs this test without actually writing any data to the drive, so let’s select “Yes, check the saved image.” Now select “Enter” to continue, and confirm that you’d like Clonezilla to create the image by entering “y” at the prompt. Clonezilla will then create the disk image and write it to /mnt/files/images on your FreeNAS machine. When complete, Clonezilla will test to make sure the image is indeed restorable and then give the option of powering off the machine, rebooting, using the command line or start over. Congratulations! You’ve just created an image of your disk and stored it on your FreeNAS machine, where it will live safely and happily until you need it.

bookmark_borderBackup FreeNAS Files Remotely Using FreeBSD and rsync

FreeNAS is an open source storage platform based on FreeBSD that supports file sharing across Windows, Apple, and Unix/Linux systems. rsync is an open source file copying utility for Linux/Unix systems. It’s famous for its “delta-transfer” algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. It offers a large number of options, including the ability to copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon; and, the ability to preserve symbolic links, hard links, file ownership, permissions, devices and times.

In this post I will describe how to configure FreeBSD, FreeNAS and rsync to securely backup files located on FreeNAS to a FreeBSD machine located at a remote location. All steps involved assume you have a running implementation of FreeBSD and FreeNAS. The software versions used in this post were as follows:

  • FreeBSD 9.0-RELEASE (x64)
  • FreeNAS v0.7.1 Shere (revision 5127)
  • rsync v3.0.9

Configure FreeNAS

There are two ways for the FreeBSD machine to contact FreeNAS using rsync: using a remote-shell program as the transport (such as ssh or rsh) or contacting an rsync daemon directly via TCP. We’re going to use ssh as our transport method. We’ll also create a new user account in FreeNAS named “rsync,” which we will enable access to via ssh but limit its use solely for the purpose of backing up files using rsync.

Begin by logging into the FreeNAS web interface, navigate to Access->Users and Groups->Users and select the “+” icon to create a new user account. The following parameters will need to be configured:

Name – This is the login name of the new user. You’re welcome to use any user name you’d like here. For purposes of our example though let’s enter rsync.

Full Name – This field should match the Name field, so enter rsync again here.

Password – Our plans include generating and using ssh keys to authenticate the FreeBSD machine to the FreeNAS machine and not passwords, so leave these fields blank.

User ID – Unless you have a specific reason to change the User ID you should retain whatever FreeNAS has chosen for you here, typically UID 1001 if this is your first new user account on FreeNAS.

Shell – This parameter specifies the login shell for our new user rsync. The nologin option should not be selected as it will not permit access to the account. For purposes of our example, we’ll select the good old sh shell.

Primary group – This parameter sets the primary group that the user rsync will belong to. Unless you have a specific reason to change the group, the default group Guest should be retained here.

Additional group – No other group memberships need be selected.

Home directory – Here’s where we’ll enter the path to the home directory for the rsync user, which for purposes of our example should be set to /usr/home/rsync.

When complete, the configuration should resemble Figure 1. Now select “Add” to accept the changes and add rsync as a new user.

Screenshot of the FreeNAS new user setup page

Figure 1

Now let’s configure and activate the rsync service in FreeNAS. Navigate to Services->Rsync->Server->Settings and check the “Enable” box, then click on the drop down list under “Map to user” and select rsync. The remaining fields can remain at their default settings. When complete, the configuration should resemble Figure 2. Now select “Save and Restart” to accept the changes and start the rsync server.

Screenshot of the FreeNAS rsync server setup page

Figure 2

Next, we need to ensure the ssh service is running in FreeNAS. Navigate to Services->SSH and check the “Enable” box. To help improve security, we’ll avoid using the default TCP port 22 and, for purposes of our example, use port 13725 instead. Check the “Permit root login” box and ensure that “Password authentication” box is checked. When complete, the configuration should resemble Figure 3. Now select “Save and Restart” to accept the changes and start the ssh server.

Screenshot of the FreeNAS SSH setup page

Figure 3

Finally, connect via ssh to the the root account of the FreeNAS machine and create the directory /usr/home/rsync/.ssh, which we’ll use to hold the public RSA key that we’ll eventually generate and copy from the FreeBSD machine. Note: Alternatively, you can create this directory using FreeNAS’s built-in file management tools by navigating to Advanced->File Manager. Make sure to provide the correct local IP address or host name of the FreeNAS machine. If this machine is located on another network, then make sure to provide the correct IP address or URL of the network where the FreeNAS machine is located, as well as ensure that any NAT gateway is configured to pass TCP port 13725 traffic to the correct host:

Okay, we’re finished with thr FreeNAS configuration for now. Let’s move on and configure the FreeBSD machine.

Configure FreeBSD

The following steps assume you have the FreeBSD Ports Collection installed. If not, you can install it by using the following commands:

If the Ports Collection is already installed, make sure to update it:

Navigate to the rsync port and build it, accepting the default configuration options:

Next, we’ll choose a location that will be used to backup the FreeNAS files. For purposes of our example, we will create the subdirectory backup within the primary user’s home directory on the FreeBSD machine:

Then make sure the user owns this directory, as well as any files and subdirectories within it:

Recall that we shunned conventional password authentication when creating the rsync user account on FreeNAS. Instead, we’re going to generate an RSA public/private key pair on the FreeBSD machine, and copy the public key to the FreeNAS machine. rsync will be able to login and authenticate itself to the FreeNAS machine without the need for a password because the FreeBSD machine has the corresponding private key. rsync can now run autonomously because it will not be prompted for a password each time it attempts to login:

By default the RSA keys are 2048 bits. If you’re really paranoid you can increase this to 4096 bits if desired with the -b flag:

You’ll be asked by the ssh-keygen script where it should store the keys (~/.ssh is the default), and then asked to enter and confirm a password for the private key that will be stored in this machine. In our particular case, we don’t want to be prompted for a password so simply press enter. The private key (id_rsa) and the public key (id_rsa.pub) are generated and then stored in ~/.ssh (if you accepted the default directory). Now that our keys are generated, let’s copy the public key to the FreeNAS machine using the venerable secure copy command:

Note that in the case of the scp command we must designate the ssh port number using the upper-case -P option, and pass to it the same ssh port number that was designated when we configured FreeNAS. Here again, make sure to provide the correct local IP address or host name of the FreeNAS machine or, if this machine is located on another network, the correct IP address or URL of the network where the FreeNAS machine is located. When you successfully connect you’ll likely receive a message concerning the authenticity of the FreeNAS machine, along with a fingerprint of its public RSA key, and asked if you’re sure you want to continue connecting. Accept by typing yes and you’ll be presented with a password prompt for the root user on the FreeNAS machine. Enter the password and the public key will be copied.

Now let’s connect again via ssh to the root account of the FreeNAS machine and copy the public key to /home/rsync/.ssh/authorized_keys so that rsync can securely connect to FreeNAS without the need for a password. We’ll also take this opportunity to remove the public key from the root account:

Then lock down permissions and make sure the user rsync owns of all files and subdirectories within its home directory:

Finally, let’s make sure we can connect to the rsync account on FreeNAS using only public/private key authentication:

When you successfully connect you’ll once again receive a message concerning the authenticity of the FreeNAS machine. Accept by typing yes and you’ll be securely connected to the rsync user’s home directory (/home/rsync/) on FreeNAS without the need to enter a password.

Okay, let’s move on and configure rsync.

Configure rsync

For purposes of example, let’s assume that on the FreeNAS machines there is directory /mnt/files/foo containing a number of files along with two subdirectories – /mnt/files/foo/bar1 and /mnt/files/foo/bar2 – that we wish to backup on a routine basis to the FreeBSD machine. You could issue the following command to the FreeBSD machine accomplish that:

Long command, right? Let’s walk through it. First, we invoke the rsync command and pass to it the following options:

-a – Specifies that the files are transferred in “archive” mode, which ensures that any symbolic links, devices, attributes, permissions, ownerships etc are preserved.

-v – Increases the verbosity of the command so that you know in a little more detail what rsync is/is not doing.

-z – Specifies that rsync should use compression to reduce the size of data portions of its file transfers.

–delete – Specifies that rsync should delete files in the destination that no longer exist on the source side. This helps to keep the file systems specified in the source and destination synchronized.

Then we tell rsync that we’d like to use ssh and port 13725 by using the “-e ssh -p 13725” option. Finally, we tell rsync what the source and destination file systems are.

Another rsync option that I find handy is –exclude. Let’s say that for whatever reason you’d like to exclude the subdirectory /mnt/files/foo/bar2 and all the files therein. You’d use the following command to accomplish that:

Please consult the rsync man pages for further information on what rsync options are available for your specific needs.

Once you get the rsync command configured the way you’d like it, then it’s time to add it to cron on the FreeBSD machine so that it will perform backups on a routine basis. cron is a *nix utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon. These tasks are typically referred to as “cron jobs.” A “crontab” is a file which contains one or more cron job entries to be run at specified times. You can create a new crontab (or edit an exiting one) using the system’s default command line editor by using the command crontab -e under your user account. Here’s some example cron job entries using our rsync command. You could add one or many rsync cron job entries to your crontab depending on your needs, then simply uncomment the one you want to use. Also, cron will attempt to email to the user the output of the commands it runs. To silence this, we’ll redirect the command output to /dev/null:

While placing the full rsync command in the crontab works just fine, it can become a bit unwieldy, particularly if the command is lengthy. A more elegant approach is to capture the command and associated options in a shell script and invoke the script in a crontab instead of the full command. Here’s what a basic shell script might look like, again utilizing our example rsync command:

You’ll notice that in addition to running the rsync command, the script will also create the log file /home/iceflatline/cronlog and append the date/time and a brief summary of what transpired to it every time script runs. To use this script, modify its values to suit your particular need, including the correct IP address or URL for the FreeNAS machine, and give it a name, say for example rsync-freenas. Then create a bin/ subdirectory in the home directory on the FreeBSD machine, copy the file to that location, and make it executable:

Now then, a crontab entry invoking this script would look like this; i.e., a whole lot less complex:

Final Steps

Now that we have everything configured, login again to the FreeNAS web interface. Navigate to Services->SSH and uncheck both the “Permit root login” and “Password authentication” boxes. When complete, select “Save and Restart” to accept the changes and restart the ssh server.

Tips

Here’s a few of tips that may help improve your experience when using rsync in the manner described in this post

  • There’s no requirement that the machine you’re backing up FreeNAS files to be based on FreeBSD. You could just as easily use a machine based on a Linux distribution for example. Simply download and install rsync via the distribution’s package manager or compile it from source. The remaining steps should work as described.
  • Regardless of which operating system you use, backing up to a separate partition or, better yet, a separate hard disk(s) is recommended. This partition or disk can simply be mounted at a location in your file system, for example the /home/iceflatline/backup we used in our example, then, if something should require you to reinstall the OS, your files would still be intact. I prefer using a separate hard disk. That way, if/when I need to restore some or all of the files to FreeNAS, I need only unmount the drive from the FreeBSD machine, mount it in FreeNAS and have access again to the files.
  • Depending on the quantity and size of the files you want to backup, you may wish to consider initially locating the FreeBSD machine on the same network as the FreeNAS machine. The initial backup will likely go much faster on a 100 Mbps or 1000 Mbps local area network than it would over a wide area network. Once the initial backup is complete, then the FreeBSD machine can be moved to a remote location to start performing incremental backups.
  • A VPN (“Virtual Private Network”) can be setup between the FreeNAS and FreeBSD machine, thus perhaps eliminating the need to use a non-standard ssh port. Another benefit of using a VPN to connect the two machines is the ability to easily connect to the FreeBSD machine from within the FreeNAS machine’s network; handy if/when you want to quickly retrieve a file or perform routine maintenance on the FreeBSD machine.
  • Occasionally, you may add large and/or numerous files to FreeNAS where the time it takes to incrementally back them up to FreeBSD exceeds the time specified between cron jobs in the crontab. For example, if the cron job runs the rsync script or command every 60 minutes, but it takes 75 minutes to incrementally backup the new files added to FreeNAS, then errors will occur when the cron job triggers rsync to run again. To avoid this problem, you can edit the crontab to lengthen the time between cron jobs, or stop the cron job from running by commenting it out in the crontab and run rsync manually instead. Once rsync has fully backed up you can use the crontab again.
  • Conclusion

    This concludes the post on how to remotely and securely backup and synchronize files located on a FreeNAS machine to a FreeBSD-based machine, using rsync, a fast and versatile file copying tool.

    References

    http://ss64.com/bash/cron.html

    http://www.marksanborn.net/linux/learning-cron-by-example/

    http://troy.jdmz.net/rsync/index.html

    bookmark_borderConfigure FreeNAS To Be Your Subversion Repository

    Recently I had the pleasure of building a Network Attached Storage (“NAS”) server based on FreeNAS. Since then, this device has more than fulfilled my initial requirements for reliable file storage and media server in my network. In a previous post, I described how I configured FreeNAS to store web files and serve as a document root for the Apache http server implemented in my Ubuntu server. Using a similar approach, this post will describe how I configured FreeNAS to host my Subversion (“svn”) repository.

    Software versions used in this post were as follows:

    • FreeNAS v0.7.1 Shere (revision 5127)
    • Ubuntu Server v10.04 LTS (x64)
    • Subversion 1.6.6dfsg-2ubuntu1
    • nfs-common v1:1.2.0-4ubuntu4
    • portmap v6.0.0-1ubuntu2

    Configuring the FreeNAS Server

    I began by creating the directory svn on /mnt/files, an existing mount point on my FreeNAS server. This directory would serve as the location for my svn repository. Then I enabled the Network File System (“NFS”) service so that /mnt/files/svn could be accessed from the Ubuntu server. To do this, navigate to Services->NFS->Settings and make sure that the check box for enabling NFS is checked and specify the number of servers that will run (the default value of four should easily handle dozens of users). Now select “Save and Restart.” Next, navigate to Services->NFS->Shares and select the “+” icon, where you are presented with the configuration screen for creating a new NFS share. Enter the path to be shared; the network that is authorized to access this shared path; and, make sure that the “All dirs” checkbox selected. The remaining options can retain their defaults (See Figure 1). Now select “Add” then “Apply changes.”

    Screenshot of NFS shared path configuration in FreeNAS

    Figure 1

    Configuring the Ubuntu Server

    In order to mount the NFS shared path without error, I needed to add a couple of packages. The nfs-common package is needed when a host acts as an NFS client, and includes a number of processes that ensure a particular NFS connection is allowed and may proceed. Also, because NFS relies upon remote procedure calls to function, the package portmap is needed in order to map RPC requests to the NFS service:

    After these requisite packages were added, I created a directory to mount the NFS shared path. In this command, you must include the IP address of the FreeNAS server as well as the path to the directory created on it previously:

    Then I made sure the permissions for this directory were set correctly:

    Next, I added the following lines to /etc/fstab in order for the shared path to mount automatically at boot time:

    Then I installed Subversion. The Subversion package for Ubuntu includes the Subversion client, tools to create a Subversion repository (svnadmin), and a custom protocol to make the repository I create on FreeNAS available over my network (svnserve):

    And created an svn repository at the root of the shared path:

    Next, I removed anonymous access to the repository and established write privileges for authenticated users. To do this, open /media/svn/conf/svnserve.conf and uncomment the following lines (Note: The svnserve.conf file is sensitive to white spaces, so make sure to not leave a space preceding the line when removing the hash (#) symbol):

    Then I added myself as an authenticated user to the /media/svn/conf/passwd file:

    I plan to use svn’s custom protocol for access to the repository so I need to start the svn server. One way to start it is to use the svnserve command:

    However, I wanted the svn server to start up at boot time, as well as have stop and restart capabilities while it was running, so I created the following simple init script:

    To use the script, I saved it to my home directory as svnserve and made it executable. Then moved it to /etc/init.d:

    Then added the svnserve script to all of Ubuntu server’s multi-user run levels (2-5) and started the svn server:

    Now, if for some reason the Ubuntu server is rebooted, the svn server will fire up automatically.

    With the svn server now running, I created a root-level directory in subversion for a project called “code”:

    After creating the project in subversion I can now use the standard svn commands or applications such Tortoise, RapidSVN and others to checkout the project or otherwise interact with the repository.

    Conclusion

    This concludes the post on how to configure a FreeNAS server for use as an svn repository. Sure, I could have installed the subversion package directly on FreeBSD, the underlying operating system for FreeNAS, but that approach adds processes that are not native to the FreeNAS implementation. By using the approach outlined in this post, I was able to place the repository on a solid, reliable and centralized storage system, and provide good logical and physical separation between the svn repository and the svn server functionality.

    References

    http://tldp.org/LDP/abs/html/abs-guide.html

    https://help.ubuntu.com/community/Subversion

    bookmark_borderConfigure FreeNAS To Store Your Apache Web Files

    (20140208 – This post has been amended to provided an updated version of the Apache directive — iceflatline)

    Over this past summer I had the pleasure of cobbling together a few spare parts in order to build a Network Attached Storage (“NAS”) box based on FreeNAS. This device has more than fulfilled my initial requirements for reliable file storage and media server in my network. This post will describe how I configured this FreeNAS box to store web files and serve as a document root for the Apache http server implemented in my Ubuntu server. This approach places my local web files on a solid, reliable and centralized RAID 5 disk storage system, and provides good logical and physical separation between file storage and file server functionality.

    The versions for the software used in this post were as follows:

    • FreeNAS v0.7.1 Shere (revision 5127)
    • Ubuntu Server v10.04 LTS (x64)
    • Apache v2.1.14
    • nfs-common v1:1.2.0-4ubuntu4
    • portmap v6.0.0-1ubuntu2

    Configuring the FreeNAS Server

    I began by creating the directory www on /mnt/files, an existing mount point. This directory would serve as my new Apache document root. Then, I enabled the Network File System (“NFS”) service in FreeNAS so that /mnt/files/www could be accessed from the Ubuntu server. To do this, navigate to Services->NFS->Settings and make sure that the check box for enabling NFS is checked and specify the number of servers that will run (the default value of four should easily handle dozens of users). Now select “Save and Restart.” Next, navigate to Services->NFS->Shares and select the “+” icon, where you are presented with the configuration screen for creating a new NFS share. Enter the path to be shared; the network that is authorized to access this shared path; and, make sure that the “All dirs” checkbox selected. The remaining options can retain their defaults (See Figure 1). Now select “Add” then “Apply changes.”

    Screenshot of NFS shared path configuration in FreeNAS

    Figure 1

    Configuring the Ubuntu Server

    To configure the Ubuntu server I needed to add a couple of packages in order to mount the NFS shared path without error. The nfs-common package is needed when a host acts as an NFS client, and includes a number of processes that ensure a particular NFS connection is allowed and may proceed. Because NFS relies upon remote procedure calls to function, the package portmap is also needed to map RPC requests to the NFS service:

    Next, I created a directory so I could mount the NFS shared path. Here you must include the IP address of the FreeNAS server as well as the directory created on it previously:

    In order for the shared path to mount automatically at boot time, I added the following lines to the /etc/fstab file:

    I made sure the directory permissions were set correctly, and modified the owner and group associated with /media/www so that the Apache http server could access it:

    Then edited the /etc/apache2/apache2.conf file, adding an alias for /media/www and a directory directive to the end of the file:

    I created an index.html file and moved it to /media/www to test the above configuration, and made sure the file permissions were set correctly:

    Finally, I restarted the Apache http server:

    Conclusion

    This concludes the post on how to configure a FreeNAS server as an Apache document root, providing a reliable way to store your web files, while at the same time utilizing the Ubuntu and Apache servers for what they do best – serving up those files.

    iceflatline