Category: BSD

Recently I decided to improve the reliability of my file system backups by using the data replication capabilities inherent in the FreeBSD Zettabyte File System (ZFS). ZFS provides a built-in serialization feature that can send a stream representation of a ZFS file system (Which ZFS refers to as a “dataset”) to standard output. Using this technique, it is possible to not only store the dataset(s) on another ZFS storage pool (zpool) connected to the local system, but also to send it over a network to another FreeBSD system. ZFS dataset snapshots serve as the basis for this replication, and the essential ZFS commands used for replicating the data are zfs send and zfs receive.

This post describes how I used this ZFS feature to perform replication of ZFS dataset snapshots from my home FreeBSD server to another FreeBSD machine located offsite. I’ll also discuss how I manage the quantity of snapshots stored locally and offsite, as well as a couple of options for recovering my files should it become necessary.

For purposes of example, I’ll refer to the FreeBSD system hosting the snapshots I want to send as “server”, and the offsite FreeBSD system that I will send snapshots to as “backup”. Unless otherwise noted, all steps were performed as the user root. However a non-root user, “iceflatline”, was created on both machines and is used for many of the commands. The versions for the software used in this post were as follows:

FreeBSD 11.0-RELEASE

Configure server

On server I had created a simple mirror vdev for my zpool consisting of (2) two terabyte disks. The mirror and the zpool were created using the following commands:

As you can see, I created one large ZFS partition (-t freebsd-zfs) on each disk. Specifying the -a option, the gpart utility tries to align the start offset and partition size on the disk to be a multiple of the alignment value. I chose 1 MiB. The advantage to this is that it is a multiple of 4096 (helpful for larger, 4 kiB sector drives), leaving the leftover fraction of a megabyte at the end of the drive. In the future, if I have to replace a failed drive containing a slightly different number of sectors, I’ll have some wiggle room in case the replacement drive is slightly larger in size. After partitioning each drive I created the zpool using these partitions. I elected to use name “pool_0” for this zpool.

To improve overall performance and usability of any datasets that I create in this zpool, I performed the following configuration changes:

The zfs command property atime controls whether the access time for files is updated when the files are read. Setting this property to off avoids producing write traffic when reading files, which can result in a gain in file system performance. The lz4 property controls the compression algorithm used for the datasets. lz4 is a high-performance replacement for the older the Lempel Ziv Jeff Bonwick (lzjb) algorithm. It features faster compression and decompression, as well as a generally higher compression ratio than lzjb. The snapdir property controls whether the directory containing my snapshots (pool_0/dataset_0/.zfs) is hidden or visible. I prefer the directory to be visible so I have another way to verify the existence of snapshots. These configuration changes were made at the zpool level so that any datasets I create in this zpool will inherit these settings; however, I could configure each dataset differently if desired.

The dataset on server that I back up offsite is called “dataset_0”, and was created using the following command:

To ensure I have still have some headroom if/when the zpool starts to get full, I set the size quota for this dataset to 80% of zpool size (1819 GiB), or 1455 GiB:

Since ZFS can send a stream representation of a dataset to standard output, it can be piped through secure shell (“SSH”) to securely send it over a network connection. By default, root user privileges are required to send and receive these streams. This requires logging into the receiving system as user root. However, logging in as the user root via a SSH is disabled by default in FreeBSD systems for security reasons. Fortunately, the necessary ZFS commands can be delegated to a non-root user on each system. The minimum delegated ZFS permissions I needed for user iceflatline to successfully send snapshots from server were as follows:

In this case I delegated the permissions at the zpool level, so any datasets I create in pool_0 will inherit them. Alternatively I could have delegated permissions at the dataset level or a combination of both if desired. There’s a lot of flexibility.

I’m able to verify which permissions were delegated anytime using the following command as either user root or iceflatline:

Finally, to avoid having to enter a password each time a backup is performed, I generated a SSH key pair as user iceflatline on server and copied the public key to /usr/home/iceflatline/.ssh/authorized_keys on backup.

Configure backup

I configured backup similar to server: a simple mirror vdev, and a zpool named pool_0 with the same configuration as the one in server. I did not create a dataset on this zpool because I will be replicating pool_0/dataset_0 on server directly to pool_0 on backup.

The minimum delegated ZFS permissions I needed for user iceflatline on backup to successfully receive these snapshots were as follows:

Using zfs send and receive

After configuring both machines it was time to test. First, I created a full snapshot of pool_0/dataset_0 on server using the following command as as user iceflatline:

While not strictly needed in this case, the -r option will recursively create snapshots of any child datasets that I may have created under pool_0/dataset_0.

Now I can send this newly created snapshot to backup, which was assigned the IP address 192.168.20.6. The following command is performed as user iceflatline:

The zfs send command creates a data stream representation of the snapshot and writes it to standard output. The standard output is then piped through SSH to securely send the snapshot to backup. The -v option will print information about the size of the stream and the time required to perform the receive operation. The -u option prevents the file system associated with the received data stream (pool_0/dataset_0 in this case) from being mounted. This was desirable as I’m using backup to simply store the dataset_0 snaphots offsite. I don’t need to mount them on that machine. The -d option is used so that all but the pool name (pool_0) of the sent snapshot is appended to pool_0 on backup. Finally, the -F option is useful for destroying snapshots on backup that do not exist on server.

zfs send can also determine the difference between two snapshots and send only the differences between the two. This saves on disk space as well as network transfer time. For example, if I perform the following command as user iceflatline:

A second snapshot pool_0/data_0@snap-test-1 is created. This second snapshot contains only the file system changes that occurred in pool_0/dataset_0 between the time I created this snapshot and the previous snapshot, pool_0/dataset_0@snap-test-0. Now, as user iceflatline, I can use zfs send with the -i option and indicate the pair of snapshots to generate an incremental stream containing only the data that has changed:

Note that sending an incremental stream will only succeed if an initial full snapshot already exists on the receiving side. I’ve also included the -R option with the zfs send command this time. This option will preserve the ZFS properties of any descendant datasets, snaphots, and clones in the stream. If the -F option is specified when this stream is received, any snapshots that exist on the receiving side that do not exist on the sending side are destroyed.

By the way, I can list all snapshots created of pool_0/dataset_0 using the following command as either user root or iceflatline:

After testing to make sure that snapshots could be successfully sent to backup, I created an ugly little script that creates a daily snapshot of pool_0/dataset_0 on server; looks for yesterday’s snapshot and, if found, sends an incremental stream containing only the file system data that has changed to backup; looks for any snapshots older than 30 days and deletes them on both server and backup; and finally, logs its output to the file /home/iceflatline/cronlog:

To use the script, I saved it to /home/iceflatline/bin with the name zfsrep.sh and, as user iceflatline, made it executable:

Then added the following cron job to the crontab under the user iceflatline account. The script runs every day at 2300 local time:

The script works is working pretty well for me, but I soon discovered that if it missed a daily snapshot or could not successfully send a daily snapshot to backup, say because either server or backup were offline or the connection between the two was down, then an error would occur the following day when the script attempts to send a new incremental snapshot. This is because backup was missing previous day’s snapshot and so the script could not send an incremental stream. To recover from this error I needed to manually send those missing snapshots. Say, for example, I had the following snapshots on server:

pool_0/dataset_0@snap-20150620
pool_0/dataset_0@snap-20150621
pool_0/dataset_0@snap-20150622

Now say that the script was not able to create pool_0/dataset_0@snap-20150623 on server because it was offline for some reason. Consequently, it was not able to successfully replicated this snapshot to backup. The next day, when server is back online, the script will successfully create another daily snapshot pool_0/dataset_0@snap-20150624 but will not be able to successfully send it to backup because pool_0/dataset_0@snap-20150623 is missing. To recover from this problem I’ll need to manually perform an incremental zfs send using pool_0/dataset_0@snap-20150622 and pool_0/dataset_0@snap-20150624:

Now both server and backup have the same snapshots and the script will function normally again.

File recovery

Having now a way to reliably replicate the file system offsite on daily basis, what happens if I need to recover some files? Fortunately, there are a couple of options available to me. First, because I chose to make snapshots visible on server, I can easily navigate to /pool_0/dataset_0/.zfs/snapshot and copy any files up to 30 days in the past (given the current retention value in the script). I could also mount pool_0/dataset_0 on backup and copy these same files from there using a utility like scp if desired.

I could also send snapshot(s) from backup to back to server. To do this I would create a new dataset on pool_0 on server. In this example, the new dataset is named receive:

Why is creating a new dataset necessary? Because there exists already the dataset pool_0/dataset_0 on server. If I tried to send pool_0/dataset_0@some-snapshot from backup back to server there would be a conflict. I could have avoided this step if I had created a dataset on pool_0 on backup and replicated snapshots of pool_0/dataset_0 to that dataset instead of directly to pool_0.

Okay, now, as user iceflatline I can send the snapshot(s) I want from backup to server:

After the stream is fully received I switch to user root and mount the dataset:

This will result in pool_0/dataset_0@snap-20150620 sent from backup to be mounted read only to pool_0/receive/dataset_0 on server. Now I can navigate to /pool_0/receive/dataset_0 and copy the files I need to recover, or I can clone or clone and promote pool_0/receive/dataset_0@snap-20150629, whatever.

Conclusion

Well, that’s it. A long and rambling post on how I’m using the replication features in FreeBSD’s ZFS to improve the reliability and resiliency of my file system backups. So far, it’s working rather well for me, and it’s been a great learning experience. Is it the best or only way? Likely not. Are there better (or at least more professional) utilities or scripts to use? Most assuredly. But for now I’ve met my most important requirement: reliably backing up my data offsite.

References

ZFS(8)
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs.html

I’ve grown tired of connecting to each host individually in my network to examine their log files. In addition to logging events locally, I would like these hosts to send their logs to a designated host in my network, resulting in a single location where I can examine and analyze all logs.

This post describes how to setup and configure a machine running FreeBSD to be a system log or “syslog” server, receiving incoming log events from other hosts in the network. A second machine, also running FreeBSD, will be configured to send its log events to the syslog server.

For purposes of example, we’ll use the hostname “server” for the machine hosting our our syslog server, and “client” for the other machine – the one sending its log events to the syslog server. All steps involved assume that FreeBSD is installed and operating correctly on both machines. All commands are issued as the root user.

The versions for the software used in this post were as follows:

• FreeBSD 11.0-RELEASE

Let’s get started…

Configure the syslog server

First, we need a file in server’s /var/log directory to host the log events coming from client. For our example, we’ll make this file name the same as client’s hostname. While you don’t need to use the .log extension, I find it helpful as it clearly indicates the purpose of the file:

Next we need to add a couple of options to syslogd, the FreeBSD utility that reads and logs messages to the system console and log files. Use sysrc to add the following line to /etc/rc.conf, substituting the IP network and network mask for your own:

The -4 (IPv4) option forces syslogd to listen for IPv4 addresses only.

The -a (allowed_peer) option specifies which clients are allowed to log to this syslog server. This option can take the form of IP address/mask:service, such as “-a 192.168.10.1/24:*” (the `*’ character permits packets sent from any UDP port), or hostname.domain, such as “-a client.home”, or “-a *.home” (assuming that the hostname can be successfully resolved to the correct IP address in the network). Multiple -a options may be also be specified. In this example, allowed_peer will the form of any host within an entire IP network, in this case 192.168.1.0/24.

Finally, the -v opton indicates verbose logging. If -v is specified once, the event’s numeric facility and priority will be added to the log. If specified more than once, the names of the event’s facility and priority (e.g., “user.notice”) are also added to the log.

Now we need to add some lines to server’s /etc/syslog.conf file, the configuration file for syslogd. First, the name of server’s hostname, preceeded by a + character, must be added to top of the file – before any existing syslog options (i.e., right before *.err; …, etc.) – so that those existing options will be applied only to log events generated by locally by server. If we did not add this line then all those options would also be applied to the log events that arrive from client. In other words, any options after a +(some_hostname) in the this file will apply until the next +(some_hostname) is parsed:

Then add following lines to the bottom of /etc/syslog.conf after the last !* , substituting the .home domain for your own:

The first line specifies that remote log events will be arriving from client. client can be specified using either its hostname or its IP address. Note that when using a hostname the domain name must be appended to it. In either case, the hostname.domain or host ip address is preceded by a + character.

The second line contains parameters to control the handling of incoming log events from client, specifically a selector field followed by an action field. The syntax of the selector field is facility.level. The facility portion describes which subsystem generated the message, such as the kernel or a daemon, while the level portion describes the severity of the event that occurred. Multiple selector fields can be used for the same action and should be separated using a semicolon (;). In our example we’ll use the * characters in the selector field to match any log events received by client.

The action field denotes where to send the log message. In our case, log events will be sent to the log file we created previously. Note that spaces are valid field separators in FreeBSD’s /etc/syslog.conf file. However, other nix-like systems still insist on using tabs as field separators. If you are sharing this file between systems, you may want to use only tabs as field separators.

Managing the log files

The file /var/log/client.log will grow over time, making it difficult to locate useful event information as well as taking up disk space. FreeBSD mitigates this problem using using newsyslog, a built-in utility that, among other things, periodically rotates and compresses log files. newsyslog is scheduled to run periodically by the system crontab (/etc/crontab). In its default configuration, it runs every hour.

newsyslog reads from its configuration file, /etc/newsyslog.conf in order to determine which actions to take. This file contains one line for each log file that newsyslog manages. Each line is comprised of various fields which control the log file’s owner and group, permissions, and when the log file should be rotated. In addition there are several optional fields for controlling log file compression and programs that should be signaled when the log file is rotated. Each field is separated with whitespace.

In order to have newsyslog recognize client’s log file, we’ll place the following line at the botton of /etc/newsyslog.conf:

In this example, the file permission for /var/log/client.log is set to 640. newsyslog will retain up to five archive files, and rotate the file when its size reaches 100 kB. The * character in the when column instructs newsyslog to ignore a time interval, a specific time, or both and instead only consider the size of the file when determining whether or not to rotate the file. The J flag tells newsyslog compress the rotated log file using bzip2, and the C flag tells newsyslog to create the log file if it does not already exist.

Finally, let’s restart syslogd and newsyslog on server:

Configure the client

Let’s move on now and configure client so that it will send its event logs to server. Open client’s /etc/syslog.conf file and add the following line after the last !*, to instruct client to send log events of any facility and level to server:

server can be specified using either its hostname, hostname.domain or its IP address, preceded by a @ character.

Now let’s restart syslogd on client:

Finally, let’s make sure client is sending its log events to server using the logger utility. Logon to client and issue the follow command:

Now login to server and and check client’s log file:

You should see the message you sent using the logger utility:

Conclusion

That’s it. In addition to logging events locally, the client host will send its logs to our syslog server, resulting in a single location where log events can be examined and analyzed.

References

NEWSYSLOG(8)
NEWSYSLOG.CONF(5)
SYSLOGD(8)
SYSLOG.CONF(5)