News Changing Code Syntax Highlighters


Most of the posts I author here contain code examples. To highlight these code examples I’ve been using the wordpress plugin SyntaxHighlighter Evolved. For the most part this plugin has served me, and I hope you, well. There are a few quirks, however, that continue to annoy me about this plugin. Chief among those quirks is the need to “minify” its Javascript and CSS following each update to maintain rendering performance. That and its lingering use of the Adobe Flash file format SWF for its copy-to-clipboard feature, even though that feature was deprecated.

So I’m making a change. I’m dropping SyntaxHighlighter Evolved for Crayon Syntax Highlighter, a nice syntax highlighter built in PHP and jQuery. It’s intuitive to administer and provides some welcome relief from those aforementioned annoyances. More importantly though it offers you some nice features. For example mousing over the highlighted code results in a drop-down box where you can toggle line numbers on/off (they’ll be on by default), toggle line wrapping on/off (longer lines will wrap by default), view plain code, copy the code, or view the code in a separate browser window. You can try these features yourself in the following example:

In the next week or so I’ll begin transitioning older posts containing code examples over to this new plugin, and of course new post will use this plugin by default. Enjoy, and please let me know if you encounter any problems.


Networking How to Install and Configure MRTG on FreeBSD


In a previous post I described how to install and configure Tobi Oetiker’s MRTG (Multi Router Traffic Grapher) on a Ubuntu server. In this post I will describe how to install and configure it on FreeBSD. Once configured, you’ll be able to use MRTG to monitor the traffic in and out of your network using the SNMP capability in your network’s gateway/router. MRTG generates static HTML pages containing PNG images which provide a visual representation of this traffic. MRTG typically produces daily, weekly, monthly, and yearly graphs. MRTG is written in perl and works on Unix/Linux as well as Windows. MRTG is free software licensed under the GNU GPL.

Software versions used in this post were as follows:

  • apache24 2.4.10_2
  • FreeBSD 10.2-RELEASE
  • mrtg-2.17.4
  • The steps discussed assume that the FreeBSD Ports Collection is installed. If not, you can install it using the following command:

    If the Ports Collection is already installed, make sure to update it:

    Okay, let’s get started. All commands are issued as user root. When building the various ports you should accept the default configuration options.

    Install a http server

    MRTG requires an http server to be installed and operating correctly. In our example, we’ll install and use the Apache http server. Navigate to the Apache port and build it:

    Once Apache has been successfully installed, add the following line to /etc/rc.conf so that the Apache server will start automatically at system boot.

    Now let’s start Apache to make sure it works:

    Point your web browser to the host name or IP address of the FreeBSD host you’ve installed Apache on and you should see the venerable “It works!”

    Install and configure MRTG

    Now that we have an http server up and running let’s install MRTG:

    What does the MRTG port install and where is that stuff located?

    MRTG provides the example configuration file /usr/local/etc/mrtg/mrtg.cfg.sample that describes global configuration parameters as well as various configuration options for the SNMP targets you want to monitor. If you already have some experience with MRTG and SNMP you can simply copy or move this file to /usr/local/etc/mrtg/mrtg.cfg then modify it to meet your requirements. In our example, however, we’re going to create the requisite mrtg.cfg file from scratch.

    MRTG includes the script cfgmaker that will create and populate a basic mrtg.cfg file with information obtained from your gateway/router. So, before running /usr/local/bin/cfgmaker, you should activiate and configure the SNMP service in your gateway/router. This typically involves logging into the device and enabling SNMP. The default SNMP community name is typically “public.” If you change the SNMP community name to something else, make note of it. Now, let’s run cfgmaker, substituting your SNMP community name if you’ve changed it, and adding the IP address of your gateway/router:

    If you would like to add more than one device to mrtg.cfg simply append the additional URL(s) to the same mrtg.cfg file. Then, when you build the web page using the indexmaker command described below, graphs associated with each device will be displayed on the same HTML page:

    Next, open /usr/local/etc/mrtg/mrtg.cfg and, under Global Config Options, uncomment the line WorkDir: /home/http/mrtg and change it to WorkDir: /usr/local/www/apache24/data/mrtg. This is the directory from which the Apache http server will server the MRTG html pages. If you’re using something other than Apache as your http server then you’ll need to change this path.

    Next, uncomment the line Options[_]: growright, bits. By default MRTG graphs grow to the left, so the option growright specifies that the direction of the traffic visible in MRTG’s graphs flips causing the current time to be at the right edge of the graph and the history values to the left. The option bits specifies that the monitored traffic values obtained from your device is multiplied by 8 and displayed bits per second instead of bytes per second.

    MRTG includes the script indexmaker. This is what we’ll use to create the pages used to display the MRTG graphs. First, let’s create the directory from which Apache http server will serve up the pages:

    Then use indexmaker combined with our mrtg.cfg file to create and populate an index.html file in that directory:

    Now we need to add an Alias and a Directory directive to Apache’s configuration file to support MRTG. Open /usr/local/etc/apache24/httpd.conf and add the following lines in the section containing similar Directory directives, or it can simply be appended to the bottom of the file:

    And change the user and group for the following directories to mrtg:

    Finally, let’s restart the http server:

    Starting MRTG

    Okay, now that MRTG has been installed and configured let’s start it up and see what it displays. Add the following line to /etc/rc.conf:

    Then start the MRTG daemon:

    The MRTG daemon will now run automatically each time FreeBSD starts.

    Now point your browser to http://your-http-server-address/mrtg and you should see a page that resembles Figure 1. You may have more or less graphs depending on the number of interfaces reported by your devices(s).

    Screenshot showing the web page generated by MRTG

    Figure 1

    You’ll see the graph starting to “grow” to the right as the traffic is monitored over time, and the Y axis displayed as bits per second. If you click on any one of these graphs you’ll be taken to differnt page showing individual graphs for 30 minute, two hour, and daily averages, along with the maximum, average, and current bit rate in and out of that particular interface. By default, these graphs will update every 5 minutes.

    Only interested in displaying one particular interface? Want to graph other SNMP data? Now that you that you have a basic mrtg.cfg file created you can modify it or incorporate some of the global and target parameter examples contained in the file /usr/local/etc/mrtg/mrtg.cfg.sample to further customize your configuration. Just remember to run indexmaker again to update the MRTG index.html file.


    This concludes the post on how to install and configure MRTG on FreeBSD. As you can see, MRTG isn’t terribly complicated and proves to be a really nice port for monitoring and graphing traffic in and out your gateway/router. For a full list of all the configuration options and other information I encourage you to visit the MRTG web site.


    Tags: ,

    BSD Using Tarsnap in FreeBSD to Improve My Offsite Backups


    In a recent post I described how I improved the reliability of my file system backups by using the data replication capabilities inherent in the FreeBSD Zettabyte File System (ZFS). In this post I will explore Tarsnap, another tool I recently started to use to perform secure offsite backups of my most important files.

    The versions for the software used in this post are as follows:

  • FreeBSD 10.1-RELEASE (running as a guest OS under VMWare ESXi hypervisor 6.0.0)
  • tarsnap 1.0.35_2
  • All commands in this post are issued as user root. The steps discussed assume that the FreeBSD Ports Collection is installed. If not, you can install it using the following command:

    Okay, let’s get started.

    Create a Tarsnap account

    Before installing Tarsnap I visited the Tarsnap registration page and created an account. Tarsnap operates on a prepaid basis, so you have to add some money to your account before you can start using it. The minimum amount is $5.00. Money will be deducted from your pre-paid amount based on the actual number of bytes stored and bandwidth used (after compression and data deduplication). Tarsnap prices are currently $.25 per Gigabyte per month for storage, and $.25 per byte for bandwidth.

    Install Tarsnap

    After creating an account it was time to install Tarsnap. First I made sure the Ports Collection was up to date:

    Then proceeded with the install, accepting all default configuration options:

    Next, I ran tarsnap-keygen, a utility which registers my machine with the Tarsnap server and generates a key that is used to encrypt and sign the archives that I create. I needed to have the e-mail address and password I used to create my Tarsnap account handy when running this command. In following example I’ve registered a machine with the host name tarsnap-test:

    Note that if I had multiple machines containing files I wished to backup to Tarsnap, I would want create a separate key file for each machine.

    By default tarsnap-keygen will create the key file /root/tarsnap.key. This can be changed by adding the option keyfile to specify a different location and/or key name. In the example above I’ve changed the name of my key file to tarsnap-test.key to help disambiguate keys in case I add additional machines to my Tarsnap account in the future.

    Tarsnap creates the file /usr/local/etc/tarsnap.conf when installed. This config file is read by the tarsnap utility and specifies a number of default options, all of which will be ignored if the options in question are specified at the command line. Since I change the name of default key file, I revised the value for the option keyfile in /usr/local/etc/tarsnap.conf:

    Note that you should store a copy of this key someplace safe. If you lose your Tarsnap key file(s), you will not be able to create new archives or access your archived data.

    Using Tarsnap

    After installing Tarsnap I was ready to create and backup my first archive. Tarsnap commands follow a syntax similar to the venerable tar utility. The -c option creates a new archive containing the specified files. The -f option specifies which file to write the archive to:

    Performing subsequent backups of these files will go faster since Tarsnap’s deduplication feature will avoid sending data which was previously stored.

    If I want to list all archives stored with Tarsnap I can use the following command:

    Adding one or more instance of the -v option to this command will make the output more verbose. For example, if -v is specified one or more times, the creation time of each archive is printed; if it is specified two or more times, the command line with which Tarsnap was invoked to create each archive is also printed.

    If I want to list the files contained within a single archive I can use the following command:

    The -t option is used to print to stdout the files stored in the specified archive; the -v option of course makes the output a little more verbose.

    If I wanted to delete one or more archives I can use the -d option:

    When the time comes to restore one or more files from Tarsnap I have a couple of options. For example, I can recover all files contained in a particular archive using the following command. In this example, I’ve extracted all files contained in the archive backup-20150729 to /tmp where I can recover one of more files:

    Or I can extract just one of the directories in this archive:

    Note here that you must exclude the leading / from the directory you’re restoring from. So in this case, instead of /pool_0/dataset_0/some-directory/, it should be pool_0/dataset_0/backup/some-directory.

    Or regress even further into the archive to recover a single file if desired:

    Finally, if for whatever reason I no longer wish to use Tarsnap on this machine I can invoke the nuke option, which will delete all of the archives stored:

    To make sure you’re really serious, Tarsnap will ask you to type the text “No Tomorrow” when using this command.

    Okay, after getting comfortable with the Tarsnap commands and backing up files manually for a couple of days, I created this ugly little script that creates a daily archive of a specified directory; looks for any archives older than 30 days and deletes them; and, logs its output to the file /home/iceflatline/cronlog:

    I wrote the script to /home/iceflatline/bin/ where I maintain some other scipts and made it executable:

    Then added the following cron job to the crontab under user root. The script runs every day at 0800 local time:

    Well, that’s it. A short post describing my experiences using Tarsnap, an easy, secure and inexpensive solution for performing offsite backups of my most important files.


    Tags: , ,

    BSD Replacing MySQL with MariaDB in FreeBSD

    1 Comment

    In my post on how to install and configure Apache, MySQL, PHP and phpMyAdmin on FreeBSD (FAMP) for basic local web development activities, one of the components is the MySQL database server. But what if you prefer to use MariaDB?

    MariaDB is an open source alternative to MySQL, and available under the terms of the GNU GPL v2 license. It is developed by the MariaDB community with oversight by the MariaDB Foundation. For all practical purposes MariaDB is a drop-in replacement for the same MySQL version. All commands, interfaces, libraries and APIs that exist in MySQL also exist in MariaDB. For example, MySQL 5.1 and MariaDB 5.1 are compatible, as are MySQL 5.5 and MariaDB 5.5. MariaDB 10.0 is the drop-in replacement for MySQL 5.6, and can also replace MySQL 5.5.

    This post will use the aforementioned post on how to install and configure Apache, MySQL, PHP and phpMyAdmin on FreeBSD as an example implementation and demonstrate how to install and configure MariaDB as a replacement for MySQL. I strongly encourage you to test these steps first before using them on your development or production environment. At the very least you should backup your database(s).

    The versions of software discussed in this post are as follows:

  • FreeBSD 10.1-RELEASE (running as a guest OS under VMWare ESXi hypervisor 6.0.0)
  • apache24 2.4.12
  • mysql56-server 5.6.24
  • mariadb55-server 5.5.43
  • mariadb100-server 10.0.17
  • mod_php56 5.6.11
  • php56 5.6.10
  • php56-extensions 1.0
  • The following steps discussed in this post assume you have the FreeBSD Ports Collection installed. If not, you can install it using the following commands:

    If the Ports Collection is already installed, make sure to update it:

    Okay, let’s get started. All commands are issued as the user root. While building the various ports you should accept all default configuration options unless otherwise instructed.

    Building FAMP with MariaDB

    To add MariaDB 10.0 when building a new FAMP implementation, use the following commands:

    Then add the following line to /etc/rc.conf:

    Start the MariaDB server:

    Then create a password for the MySQL root user:

    If you’d like to add MariaDB 5.5 instead of MariaDB 10.0 when building a new FAMP implementation, use the following commands:

    Then use the commands described previously to add MariaDB to /etc/rc.conf, start the server, and create the root password.

    Replacing MySQL with MariaDB in an existing FAMP implementation

    If you’ve previously built FAMP using the MySQL 5.6 server, then you can replace it with MariaDB 10.0. First, make sure to backup your database(s), then uninstall the MySQL 5.6 server and client:

    Install MariaDB 10.0:

    Then finish by running the command mysql_upgrade. This command does two things: it ensures that your mysql privilege and event tables are updated with the new fields MariaDB uses; and, it performs a check of all tables and marks them as compatible with MariaDB 10.0. In most cases this should be a fast operation (depending on the number of database tables):

    If you’re currently using MySQL 5.5, then you can replace it with either MariaDB 5.5 or MariaDB 10.0. To replace it with MariaDB 5.5:

    While not strictly necessary in this case, you should also run mysql_upgrade. If calling mysql_upgrade was not necessary, it does nothing.


    Well, that’s it. A few minutes of your time with the FreeBSD Ports Collection and you can quickly replace MySQL with the open source alternative MariaDB on your FAMP implementation.


    Tags: ,

    BSD Using ZFS replication features in FreeBSD to improve my offsite backups

    1 Comment

    Recently I decided to improve the reliability of my file system backups by using the data replication capabilities inherent in the FreeBSD Zettabyte File System (ZFS). ZFS provides a built-in serialization feature that can send a stream representation of a ZFS file system (Which ZFS refers to as a “dataset”) to standard output. Using this technique, it is possible to not only store the dataset(s) on another ZFS storage pool (zpool) connected to the local system, but also to send it over a network to another FreeBSD system. ZFS dataset snapshots serve as the basis for this replication, and the essential ZFS commands used for replicating the data are zfs send and zfs receive.

    This post describes how I used this ZFS feature to perform replication of ZFS dataset snapshots from my home FreeBSD server to another FreeBSD machine located offsite. I’ll also discuss how I manage the quantity of snapshots stored locally and offsite, as well as a couple of options for recovering my files should it become necessary.

    For purposes of example, I’ll refer to the FreeBSD system hosting the snapshots I want to send as “server”, and the offsite FreeBSD system that I will send snapshots to as “backup”. Unless otherwise noted, all steps were performed as the user root. However a non-root user, “iceflatline”, was created on both machines and is used for many of the commands. The versions for the software used in this post were as follows:

  • FreeBSD 10.1-RELEASE
  • Configure server

    On server I had created a simple mirror vdev for my zpool consisting of (2) two terabyte disks. The mirror and the zpool were created using the following commands:

    As you can see, I created one large ZFS partition (-t freebsd-zfs) on each disk. Specifying the -a option, the gpart utility tries to align the start offset and partition size on the disk to be a multiple of the alignment value. I chose 1 MiB. The advantage to this is that it is a multiple of 4096 (helpful for larger, 4 kiB sector drives), leaving the leftover fraction of a megabyte at the end of the drive. In the future, if I have to replace a failed drive containing a slightly different number of sectors, I’ll have some wiggle room in case the replacement drive is slightly larger in size. After partitioning each drive I created the zpool using these partitions. I elected to use name “pool_0” for this zpool.

    To improve overall performance and usability of any datasets that I create in this zpool, I performed the following configuration changes:

    The zfs command property atime controls whether the access time for files is updated when the files are read. Setting this property to off avoids producing write traffic when reading files, which can result in a gain in file system performance. The lz4 property controls the compression algorithm used for the datasets. lz4 is a high-performance replacement for the older the Lempel Ziv Jeff Bonwick (lzjb) algorithm. It features faster compression and decompression, as well as a generally higher compression ratio than lzjb. The snapdir property controls whether the directory containing my snapshots (pool_0/dataset_0/.zfs) is hidden or visible. I prefer the directory to be visible so I have another way to verify the existence of snapshots. These configuration changes were made at the zpool level so that any datasets I create in this zpool will inherit these settings; however, I could configure each dataset differently if desired.

    The dataset on server that I back up offsite is called “dataset_0”, and was created using the following command:

    To ensure I have still have some headroom if/when the zpool starts to get full, I set the size quota for this dataset to 80% of zpool size (1819 GiB), or 1455 GiB:

    Since ZFS can send a stream representation of a dataset to standard output, it can be piped through secure shell (“SSH”) to securely send it over a network connection. By default, root user privileges are required to send and receive these streams. This requires logging into the receiving system as user root. However, logging in as the user root via a SSH is disabled by default in FreeBSD systems for security reasons. Fortunately, the necessary ZFS commands can be delegated to a non-root user on each system. The minimum delegated ZFS permissions I needed for user iceflatline to successfully send snapshots from server were as follows:

    In this case I delegated the permissions at the zpool level, so any datasets I create in pool_0 will inherit them. Alternatively I could have delegated permissions at the dataset level or a combination of both if desired. There’s a lot of flexibility.

    I’m able to verify which permissions were delegated anytime using the following command as either user root or iceflatline:

    Finally, to avoid having to enter a password each time a backup is performed, I generated a SSH key pair as user iceflatline on server and copied the public key to /usr/home/iceflatline/.ssh/authorized_keys on backup.

    Configure backup

    I configured backup similar to server: a simple mirror vdev, and a zpool named pool_0 with the same configuration as the one in server. I did not create a dataset on this zpool because I will be replicating pool_0/dataset_0 on server directly to pool_0 on backup.

    The minimum delegated ZFS permissions I needed for user iceflatline on backup to successfully receive these snapshots were as follows:

    Using zfs send and receive

    After configuring both machines it was time to test. First, I created a full snapshot of pool_0/dataset_0 on server using the following command as as user iceflatline:

    While not strictly needed in this case, the -r option will recursively create snapshots of any child datasets that I may have created under pool_0/dataset_0.

    Now I can send this newly created snapshot to backup, which was assigned the IP address The following command is performed as user iceflatline:

    The zfs send command creates a data stream representation of the snapshot and writes it to standard output. The standard output is then piped through SSH to securely send the snapshot to backup. The -v option will print information about the size of the stream and the time required to perform the receive operation. The -u option prevents the file system associated with the received data stream (pool_0/dataset_0 in this case) from being mounted. This was desirable as I’m using backup to simply store the dataset_0 snaphots offsite. I don’t need to mount them on that machine. The -d option is used so that all but the pool name (pool_0) of the sent snapshot is appended to pool_0 on backup. Finally, the -F option is useful for destroying snapshots on backup that do not exist on server.

    zfs send can also determine the difference between two snapshots and send only the differences between the two. This saves on disk space as well as network transfer time. For example, if I perform the following command as user iceflatline:

    A second snapshot pool_0/data_0@snap-test-1 is created. This second snapshot contains only the file system changes that occurred in pool_0/dataset_0 between the time I created this snapshot and the previous snapshot, pool_0/dataset_0@snap-test-0. Now, as user iceflatline, I can use zfs send with the -i option and indicate the pair of snapshots to generate an incremental stream containing only the data that has changed:

    Note that sending an incremental stream will only succeed if an initial full snapshot already exists on the receiving side. I’ve also included the -R option with the zfs send command this time. This option will preserve the ZFS properties of any descendant datasets, snaphots, and clones in the stream. If the -F option is specified when this stream is received, any snapshots that exist on the receiving side that do not exist on the sending side are destroyed.

    By the way, I can list all snapshots created of pool_0/dataset_0 using the following command as either user root or iceflatline:

    After testing to make sure that snapshots could be successfully sent to backup, I created an ugly little script that creates a daily snapshot of pool_0/dataset_0 on server; looks for yesterday’s snapshot and, if found, sends an incremental stream containing only the file system data that has changed to backup; looks for any snapshots older than 30 days and deletes them on both server and backup; and finally, logs its output to the file /home/iceflatline/cronlog:

    To use the script, I saved it to /home/iceflatline/bin with the name and, as user iceflatline, made it executable:

    Then added the following cron job to the crontab under the user iceflatline account. The script runs every day at 2300 local time:

    The script works is working pretty well for me, but I soon discovered that if it missed a daily snapshot or could not successfully send a daily snapshot to backup, say because either server or backup were offline or the connection between the two was down, then an error would occur the following day when the script attempts to send a new incremental snapshot. This is because backup was missing previous day’s snapshot and so the script could not send an incremental stream. To recover from this error I needed to manually send those missing snapshots. Say, for example, I had the following snapshots on server:


    Now say that the script was not able to create pool_0/dataset_0@snap-20150623 on server because it was offline for some reason. Consequently, it was not able to successfully replicated this snapshot to backup. The next day, when server is back online, the script will successfully create another daily snapshot pool_0/dataset_0@snap-20150624 but will not be able to successfully send it to backup because pool_0/dataset_0@snap-20150623 is missing. To recover from this problem I’ll need to manually perform an incremental zfs send using pool_0/dataset_0@snap-20150622 and pool_0/dataset_0@snap-20150624:

    Now both server and backup have the same snapshots and the script will function normally again.

    File recovery

    Having now a way to reliably replicate the file system offsite on daily basis, what happens if I need to recover some files? Fortunately, there are a couple of options available to me. First, because I chose to make snapshots visible on server, I can easily navigate to /pool_0/dataset_0/.zfs/snapshot and copy any files up to 30 days in the past (given the current retention value in the script). I could also mount pool_0/dataset_0 on backup and copy these same files from there using a utility like scp if desired.

    I could also send snapshot(s) from backup to back to server. To do this I would create a new dataset on pool_0 on server. In this example, the new dataset is named receive:

    Why is creating a new dataset necessary? Because there exists already the dataset pool_0/dataset_0 on server. If I tried to send pool_0/dataset_0@some-snapshot from backup back to server there would be a conflict. I could have avoided this step if I had created a dataset on pool_0 on backup and replicated snapshots of pool_0/dataset_0 to that dataset instead of directly to pool_0.

    Okay, now, as user iceflatline I can send the snapshot(s) I want from backup to server:

    After the stream is fully received I switch to user root and mount the dataset:

    This will result in pool_0/dataset_0@snap-20150620 sent from backup to be mounted read only to pool_0/receive/dataset_0 on server. Now I can navigate to /pool_0/receive/dataset_0 and copy the files I need to recover, or I can clone or clone and promote pool_0/receive/dataset_0@snap-20150629, whatever.


    Well, that’s it. A long and rambling post on how I’m using the replication features in FreeBSD’s ZFS to improve the reliability and resiliency of my file system backups. So far, it’s working rather well for me, and it’s been a great learning experience. Is it the best or only way? Likely not. Are there better (or at least more professional) utilities or scripts to use? Most assuredly. But for now I’ve met my most important requirement: reliably backing up my data offsite.



    Tags: , ,