Print-friendly Version

Beginner's Guide to Tape Drive Performance Tuning on Solaris 10 Systems

Vincent Esposito, May 2008

Introduction

This article provides tips and examples for how to tune the performance of tape drives by determining the optimal block size to use during backups and restores. It covers the following topics:

• Background
• Test-Bed Specifications
• Baseline Measurements
• Source of the Problem: Default Blocking Factors
• Adjusting the Block Size
• Patterns: Restores Take Longer Than Backups
• Determining Optimum Block Size
• First Round of Tests
• Second Round of Tests
• Conclusion: Putting Optimal Block Size to Use
• About the Author

Background

Tuning the performance of tape drives on systems that run the Solaris Operating System is a frequently overlooked topic. Many system administrators get the latest and greatest tape technology that's available and add it to their backup environment. Then, when the backups run "fast enough" to fit in the available backup time window, they declare the system to be better than it was before. Many simply look at the tape drive vendor's published statistics and assume that the devices work at those speeds all the time.

Doing this is a recipe for disaster or at least for major headaches down the road. The backup requirements might gradually (or in some cases, explosively) grow to the point where backups no can no longer be completed without encroaching on time periods that need to be dedicated to production. Worse yet, it might be time for a disaster recovery exercise, or there might even be an actual disaster that requires restoring huge amounts of data.

These situations are when most system administrators discover that their tape drives are not running as fast as the published specs. More frequently, a system administrator discovers that the backups run "fast enough," but restores take an amount of time that defies even the most conservative service-level agreement regarding recovery-time objectives. Many cry foul! Restores refuse to run at even a fraction of the vendor's stated data transfer rates.

Unfortunately, it is not the vendor's fault. Although the vendor might use data, disk subsystems, and servers all carefully tuned to produce the highest possible numbers in official benchmarks, the devices are actually capable of the published rates, for backups at least. In general, the default settings for most native commands are woefully out of date with current tape technology.

Where tape drives are concerned, a little research and experimentation can go a long way. A huge number of factors can affect the performance of backups and restores, ranging from the disk subsystem used to physical memory amounts, shared memory settings, and even logical volume management software. However, the primary tuning strategy that can give the greatest gains (or losses!) in performance is simply adjusting the blocking factor used when writing and reading.

Experimenting with adjusted blocking using enterprise backup software products can be difficult, though. Each software vendor has its own methods for tuning that might or might not be easily adjustable, and it might be impractical to experiment within a production environment. A more suitable method is to use tools native to the Solaris OS to get a feel for the tuning values that work in your environment, and then consult your backup software vendor's documentation or support personnel for help making the adjustments indicated by your findings using whatever modifications are required to transition from the Solaris native commands to the enterprise backup software.

Test-Bed Specifications

The later sections of this article quote numbers generated on an actual test-bed system. In order to make a strong point, we used a fairly fast, modern tape drive: the IBM 3580L33 Ultrium 3 tape drive (also known as LTO-3, where "LTO" stands for "Linear Tape-Open"). This was attached as /dev/rmt/1 to a Sun StorEdge PCI Dual Ultra3 SCSI Host Adapter on a relatively fast server based on SPARC technology. (A Sun Fire v880 server with 8x900MHz CPUs and 32 Gbytes of physical memory running the Solaris 10 08/07 OS.)

For test data, we created a 2-Gbyte junk file under /mnt using the mkfile command. While this data is not necessarily indicative of a real-world data set, it is enough to generate meaningful performance data and demonstrate the basic concepts of tape drive tuning.

Baseline Measurements

The tests were run using two basic Solaris commands to repeatedly back up and restore the 2-Gbyte junk file: tar and dd. To get performance statistics, we can use the time command, for example:

# time tar cvf /dev/rmt/1 /mnt/junk

This command writes the file /mnt/junk in tar format to the /dev/rmt/1 tape drive and returns the total amount of time for the operation. It also returns User and Sys time, which indicate time spent waiting on input or waiting for system operations to be completed, respectively. While somewhat crude, time is accurate enough to generate meaningful performance numbers that can help determine a reasonable range of tuning values.

A simple calculation gives us the "speed" of the command in Mbytes/sec. We know that we are moving 2 Gbytes of data (2048 Mbytes), and from the time command, we can find out how many seconds the command ran. Simply dividing 2048 by the number listed in the time command's output for Real (converted into seconds) gives us our data rate.

For example, if we run a command that time reports as taking 34 minutes and 8 seconds (2048 seconds), we know that we've moved data at a rate of 2048 Mbyte / 2048 seconds = 1 Mbyte/sec.

To generate a few baseline numbers, we back up and restore the 2-Gbyte /mnt/junk file using tar with the default blocking factor, as follows.

Backup:

# time tar cvf /dev/rmt/1 /mnt/junk

Real:	2:13.7
User:	.6
Sys:	10

This gives us a total runtime of 2 minutes and 13.7 seconds (133.7 seconds). Using our formula, our base tar backup ran at 15.32 Mbyte/sec, which is far below the published, native speed for LTO-3 technology (80Mbyte/sec).

Restore:

# time tar xvf /dev/rmt/1

Real: 2:07.1
User: 11
Sys:  1:02.3

This translates to a runtime of 127.1 seconds at 16.11 Mbyte/sec. Again, this is not very inspiring considering the published throughput for LTO-3 technology.

We can try the dd command with default settings as well to see how that compares:

Backup:

# time dd if=/mnt/junk of=/dev/rmt/1

Real:  15:25.0
User:  23.5
Sys:   1:52.0

This time, the backup took an extraordinary 15 minutes and 25 seconds (925 seconds) at 2.21 Mbyte/sec. This appears to be another discouraging discovery.

Restore:

# time dd if=/dev/rmt/1 of=/mnt/junk

Real:	17:30.5
User:	32
Sys:	2:59.0

The restore runs for 17 minutes and 30.5 seconds (1050.5 seconds) at 1.95 Mbyte/sec. This number is about 1/40th of the tape drive's published native throughput.

Source of the Problem: Default Blocking Factors

The trick here is that both the tar and dd commands were written many years ago when a tape drive that could run at these speeds was considered fast. The key is the blocking factor (or block size) used. This value basically acts as a buffer that holds a chunk of the data being written to or read from tape. Tuning this block size allows us to choose a size that ensures that the tape drive has enough data to keep it busy until the next chunk of data is ready to be written. Conversely, during a restore, the buffer is sized so that the tape drive can fill this buffer quickly enough to keep the system's disks busy.

The tar command uses a blocking factor that is an integer value that can range from 1 up to a maximum defined by the system's available memory. According to the man page for tar, each factor is equivalent to 512 bytes, and if given no options, tar assumes a default blocking factor of 20 (or 10240 bytes = 10 Kbyte). This default value was chosen a very long time ago by a programmer who most likely had access to only old quarter-inch streaming tape. This default value was chosen through trial and error and experimentation to be suitable for the majority of the tape technology of the time, and while the value has most likely been updated since, these updates have not kept full pace with current tape technology. In short, the default blocking factor value for tar is far too low to work reasonably with most modern tape technology, such as LTO-3 technology.

The dd command is a product of an even older generation, and it is probably one of the oldest I/O commands in UNIX. According to the man page for dd, dd assumes a block size of 512 bytes by default (20 times smaller than tar does!). A value of 512 bytes was chosen mainly because this was the native block size used by most devices of the era, especially quarter-inch streaming tape. So again, we have a native command tuned for out-of-date technology. The dd command does have one main advantage over tar, though. Its block size can be tuned to any arbitrary size, not just multiples of 512 bytes.

The blocking factor, or block size, can be adjusted to optimize backup speeds. However, adjusting the blocking factor has repercussions when you run restores. The blocking factor, or block size, used to make the backup must be used when the restore is run, or else performance might suffer and, in some cases, cause the restore to fail.

The tar command has a useful feature. If the blocking factor specified on a restore is larger than the blocking factor used to make the tape, tar is able to adjust downward. If a restore is attempted using a number lower than the blocking factor of the backup, however, tar fails with a read error. Understanding this behavior is very useful when you are handed a tape, told it is in tar format, and told you need to pull data off the tape, but you are given no clues regarding the blocking factor. In such a case, you could simply choose an arbitrarily large blocking factor if the default value fails, and increase the value until the restore can run.

The dd command offers no such pleasantries. If you do not know the blocking factor, you are in for a long day at the office.

Adjusting the Block Size

To demonstrate the effectiveness of an adjusted blocking factor, let's run our backup and restore again using 256-Kbyte blocks, which, at first, appears incredulously large compared to the default values. Keep in mind just how far out of date the default values are, and let the numbers be your guide. Remember that for the tar command, we will use a value that represents 256 Kbytes divided by 512 bytes (this number happens to be 512, since 512 * 512 = 256 Kbytes), and for the dd command, we can use the actual number expressed in kilobytes (256k).

Backup using tar:

# time tar cvfb /dev/rmt/1 512 /mnt/junk

Real:  28.1
User;  0.0
Sys:   3.5

This gives us 2048 Mbyte / 28.1 seconds = 72.88 Mbyte/sec, which is a number comparable to the published statistics for an LTO-3 tape.

Restore using tar:

# time tar xvfb /dev/rmt/1 512

Real: 1:23.2
User:  10.5
Sys:   56.2

This yields 2048 Mbyte / 83.2 seconds = 24.62 Mbyte/sec, which is not as high as we would like to see, but far better than our original numbers.

Backup using dd with the block size (bs) specified as 256k:

# time dd if=/mnt/junk of=/dev/rmt/1 bs=256k

Real: 28.0
User: 0.0
Sys:  3.5

These results yield 2048 Mbyte / 28 seconds = 73.14 Mbyte/sec.

Restore using dd with block size specified as 256k:

# time dd if=/dev/rmt/1 of=/mnt/junk bs=256k

Real: 36.5
User: 0.0
Sys:  7.5

This yields 2048 Mbyte / 36.5 seconds = 56.11 Mbyte/sec, which is our best speed on a restore so far.

Patterns: Restores Take Longer Than Backups

If we examine the breakdown of numbers reported by time in each case, we see an additional pattern. Clearly restores appear to run significantly slower than backups in the majority of cases. Restores done using tar also appear to have significantly higher User and Sys values, indicating a larger processing overhead cost to the operation. In our best cases, the restores appear to run about 70% as fast as the backups (when using dd with the block size specified as 256k) down to about 30% as fast (when using tar with a blocking factor of 512). This clearly demonstrates that even when tuning is performed that greatly increases the backup speed, restores still run significantly slower than the backups.

Many system administrators fall into the trap of believing that they can expect restores to run at speeds comparable to their backups, and they end up spending a lot of time trying to figure out why their restores are slow when they suddenly need to recover a large data set, or they are forced to re-examine their backup and recovery strategies when they suddenly discover that they are unable to meet their time commitments for system recovery. Never assume that your restore speed will be close to your backup speed without thoroughly testing first!

Determining Optimum Block Size

We now know that newer, faster tape drives begin to achieve their true potential only when we tune block size appropriately; however, how do we determine the actual optimum value to use? There is no simple answer to this question when you consider the vast number of factors that can affect performance.

One method is to take a known quantity of data on a server that replicates, as closely as possible, the production environment (including number of CPUs, CPU speeds, memory, swap, shared memory settings, disk types, RAID levels, and so on). Then run several test cases at different block sizes, and time the results to home in on what appears to be the best value. Most modern tape drives can actually take in data faster than a single disk can produce, so you might find that backup speeds level off at certain points unless you are using a RAID setup that allows multiple disks to stream data at the same time.

The way you decide which values to work with is also determined by your production priorities. You might make backups your priority and be willing to sacrifice some speed on the restore side in order to fit your backups into a tight nightly window. You might need to focus more on getting the most performance out of restores to reduce turnaround time for large-scale data set recovery and be willing to accept that your backups might take a little longer. In most cases, you'll want to strike a balance between the two, which is the most sensible approach unless your environment has very specific needs.

First Round of Tests

The test data we will work with in our next examples might appear to follow very different patterns than what you'll see in your production environment. You might have faster or slower disks, different kernel tuning parameters on different servers, different SCSI adapters, and a whole host of other minor differences that might generate a different performance profile. The examples shown are for demonstration purposes only. We will track both backups and restores using dd at a variety of block sizes and get a feel for the general pattern as we increase values. We might find that as block size increases, performance increases but by progressively smaller increments, or we might find that we have a bell curve where speed increases with block size up to a certain point, and then tapers off. We might also see that there can be more than one peak performance number, and we might see the pattern differs for backups and restores. These patterns will be very individual depending on the specific details of each system, so again, it should be stressed that it is important to replicate the circumstances you want to optimize performance for as closely as possible. Also, realize that optimal settings on one server, or for one set of backups and restores, might not be quite as optimal for others.

We will test by using the following command for backups, where <X> is the block size:

# time dd if=/mnt/junk of=/dev/rmt/1 bs=<X>

And we will test using the following command for restores:

# time dd if=/dev/rmt/1 of=/mnt/junk bs=<X>

It should be noted that you should delete your "junk" file after each backup before running the restore, and recreate it using mkfile after running each restore. Doing this compensates for the tendency of dd, and the hardware compression on the tape drive, to inadvertently alter the final size of the file. Since this file is basically a fixed bit pattern normally used to create swap files, it tends to accumulate strange side effects when it is run through block size changes and compression algorithms not normally seen with normal production data.

We will use a very broad range of values so we can see if there is a pattern to the data and get an idea of what block size values we would like to investigate further. Keep in mind that we are running a single backup and a single restore at each block size, so some numbers might be statistical anomalies in this run. Once we have this baseline to work from, we will choose a few values to focus on and get more statistically significant data.

After running tests specifying bs as 16k, 32k, 64k, 128k, 256k, 512k, 1024k, 2048k, 4096k, and 8192k (basically increasing by powers of two), we get data that looks like the following.

Peak performance for backups appears to occur with block size specified as 2048k, while peak performance for restores appears to occur with 512k. Our data, in this case, seems to indicate a rough bell curve as we increase block size values on both the backup and the restore side; however, each seems to peak at different points. Again, since we have taken only one set of timings for each data point, we might not have entirely consistent information. Note the very high restore speed at the 64k mark. It is unlikely that this represents the average rate at this block size, since the number does not seem consistent with the rest of the data.

To home in further, we will take a closer look at the values around the peak numbers we observed. We will do several test runs at each block size level, average them, and then use the average, rather than a single data point, to base our decisions on. In our example, we will run our test five times for each block size. In a real production scenario, the more test runs you perform for each block size, the more accurate your final number will be. It would probably be a good idea to run at least 10 tests or more for each data point.

We will use block size values of 256k, 512k, 1024k, and 2048k for our more detailed tests.

From the data generated in our test runs, we can see that the five test averages are all very close. The backup speed at a block size of 2048k appears to be the best of the group by a small margin, and the restores at all the tested block sizes appear to be less than .5 Mbyte/sec apart.

The data seems to indicate that the optimal values lie somewhere within this range of numbers. If you were optimizing for backups, you'd probably prefer to lean towards the 2048k block size, and basically any of these numbers work for restores, although you might prefer to favor a block size that has the most consistent restore time across all test runs.

If we look closely at the restore times, it appears that the restores at a 512k block size appear to be the most consistent across all five runs.

While our data has pointed us in the right direction, it should not be considered an exhaustive test. In a real-world situation, we would probably want to take a closer look at average runs across this range of numbers, and increment our block sizes by smaller amounts until we feel that we've reached a satisfactory level of detail. A good practice would be to use a general, wide range of numbers as we did previously, and then do a more detailed breakdown of all the possibilities in that range of numbers.

If we wanted to continue in further detail, we could take averages of 10 test runs using block sizes starting with 256k and going up by 64k increments until we reach 1024k.

Of course, these numbers do not tell us everything. There might be a bottleneck somewhere placing an artificial limit on our data transfer rate. The hard disk itself might be the bottleneck, and it might not be able to produce data as fast as the tape drive is capable of taking data (and conversely, the hard drive might not be able to take in data as fast as the tape drive can deliver data during restores). A useful test for confirming the optimal block size we determined earlier is to take the disk subsystem out of the picture. We can use the system device /dev/zero, which is a register that constantly spits out bytes made up of eight zeros, instead of a disk-bound test file.

Using /dev/zero also has another advantage: A constant stream of a simple bit pattern is the easiest data set for the tape drive's hardware-based compression to process. A string of nothing but zeros is the simplest pattern in the world, and even the most rudimentary compression routine can handle it at maximum efficiency.

Second Round of Tests

For our next round of tests, we will need a different method for determining the throughput rate. Since /dev/zero spits out a constant stream, any backups we run will never actually finish until we hit end-of-tape, unless we interrupt the process. So, using the time command is impossible. Also, /dev/zero effectively is infinite in size, so we have no way of using time to capture a runtime, nor do we have a fixed amount of data to use as a baseline. We can, instead, use the iostat command in a separate window to view the throughput in real time while we run our dd commands. For ease of readability, we can use the -Mnx options with an interval of 1 to view a snapshot of I/O activity every 1 second, display throughput in Mbyte/sec, and display device names by logical name rather than by instance number, as follows:

# iostat -Mnx 1

For our backup operations, we will use the following command, where <X> is our block size:

# dd if=/dev/zero of=/dev/rmt/1 bs=<X>

This command streams /dev/zero to our tape drive at the block size we choose. It runs indefinitely, so we need to watch the iostat output long enough for the tape drive to spin up to full speed, and read off the Mw/sec (Mbytes written per second) field and the %b (percentage busy) field. About 10 to 20 seconds should be enough, and then we can interrupt the command by pressing Ctrl-C.

We can rewind, and then run a "simulated" restore by dumping the /dev/zero byte stream we've captured on the tape to /dev/null (basically, restore the data to nowhere) using the following command, where <X> is the block size:

# dd if=/dev/rmt/1 of=/dev/null bs=<X>

Again, we need to watch the iostat output, and observe the Mr/sec field (Mbytes read per second) and the %b (percent busy) field. We'll want to run this test on a range of block size values as we did earlier: 16k, 32k, 64k, 128k, and so on up to 8192k, and then look at our results.

We see what appears to be an amazingly high restore throughput that actually far exceeds the backup throughput. This is, in part, an artifact of the simplistic data set we used, and the fact that the drive's hardware compression algorithm can handle it in an optimal manner. We will probably never see sustained throughput like this in a real-world backup or restore (unless we are multiplexing multiple backups from multiple disks to the same tape drive).

When we look at these numbers, we get a better sense of how manufacturers benchmark their tape devices by using raw data that can be compressed with maximum efficiency and streamed in or out as fast as (or faster than) the tape drive.

In this round of tests, however, we are more concerned with the block size that generates the best numbers on both backups and restores. As before, we see that the data seems to describe a rough bell curve: gradually rising until a peak is hit and then tapering off after the peak is passed.

The peak of the curve for both backups and restores appears to be at the 2048k block size, which is consistent with our findings earlier. We can clearly see that the transfer rates achieved here are much higher than when we were reading from and writing to disk, which indicates that the speed of the disk subsystem is a definite limiting factor.

Conclusion: Putting Optimal Block Size to Use

So, now that we know what appears to be the optimal block size to use, what do we do with this information? The answer to that is highly dependent on what backup solution you use. If you are using native utilities, such as tar, dd, ufsdump, and so on, it is a simple matter of translating this number into a blocking factor meaningful to whichever command sets you use for your backups and restores.

If you are using an enterprise backup solution, such as Symantec Veritas NetBackup, IBM Tivoli Storage Manager, or EMC Legato Networker, you probably need to do further research in order to apply this knowledge to your backup and recovery software. Most major vendors offer some mechanism for tuning buffer sizes or blocking factors, but this might be buried in hard-to-find places in the official documentation.

Each software package might have its own limits on how far you can tune the buffer or block size, in which case you will want to get as close as possible to the recommended values suggested by your tests.

Additionally, altering buffer or blocking values might also require additional related adjustments, and it might even require adjusting kernel tunable parameters to ensure that enough shared memory is allocated. The exact nature of the task ahead of you will vary from vendor to vendor, and it will vary for different environments as well. It is highly recommended that you research tuning your enterprise backup software thoroughly before proceeding, and test the results of any changes you make.

In addition to the official vendor documentation, you might be able to find resources on the Internet that describe how other people tuned their systems. These resources can be invaluable, because they allow you to learn from someone else's mistakes rather than your own. If you feel overwhelmed or lost while going through this process, a service call to the software vendor might be the best way to clear things up, provided you have a service contract.

Tuning is by no means an easy task, and we have only scratched the surface here. I hope you have taken away a few valuable concepts and insights that will help you start down the road to increased throughput.

About the Author

Vincent has worked with the Solaris OS in various capacities since 1994. He has worn a variety of job titles and functions over the intervening years, including tape operator, help desk guy, backup administrator, hardware technician, system administrator, and disaster recovery technician. In his current capacity, he is primarily responsible for providing support to, and developing infrastructure in support of, disaster recovery exercises. He also tries to devote as much time as possible to a wide range of minor research projects and scripting projects, as well as writing documentation. His specialties include JumpStart, backup and recovery, enterprise-level backup solutions, server deployment, and stashing spare parts in the bottom left drawer of his desk in case he needs them in an emergency.

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.