Benchmark disk IO with DD and Bonnie++

Benchmark disk IO with DD and Bonnie++

Get Social!

Benchmarking disk or file system IO performance can be tricky at best. The problem is that modern file systems leverage various techniques to ensure that the best performance is achieved such as caching files in RAM. This means that unless you circumvent the disk cache, your reported speeds will be reporting how quickly the files can be read from memory.

In this example, I’ll cover benchmarking a Linux file system using two methods; dd for the easy route, and bonnie++ for a more comprehensive test.

dd

Write

You can use dd to create a large file as quickly as possible to see how long it takes. It’s a very basic test and not very customisable however it will give you a sense of the performance of the file system. You must make sure this file is larger than the amount of RAM you have on your system to avoid the whole file being cached in memory.

It’s usually installed out-of-the-box with most Linux file systems which makes it an ideal tool in locked-down environments or environments where it’s tricky to get packages installed onto. Use the below command substituting [PATH] with the filesystem path to test, [BLOCK_SIZE] with the block size and [LOOPS] for the amount of blocks to write.

time sh -c "dd if=/dev/zero of=[PATH] bs=[BLOCK_SIZE]k count=[LOOPS] && sync"

A break down of the command is as follows:

  • time – times the overall process from start to finish
  • of= this is the path which you would like to test. The path must be read/ writable.
  • bs= is the block size to use. If you have a specific load which you are testing for, make this value mirror the write size which you would expect.
  • sync – forces the process to write the entire file to disk before completing. Note, that dd will return before completing but the time command will not, therefore the time output will include the sync to disk.

The below example uses a 4K block size and loops 2000000 times. The resulting write size will be around 7.6GB.

time sh -c "dd if=/dev/zero of=/mnt/mount1/test.tmp bs=4k count=2000000 && sync"
2000000+0 records in
2000000+0 records out
8192000000 bytes transferred in 159.062003 secs (51501929 bytes/sec)
real 2m41.618s
user 0m0.630s
sys 0m14.998s

Now, let’s do the math. dd tells us how many bytes were written, and the time command tells us how long it took – use the real output at the bottom of the output. Use the formula BYTES / SECONDS. For these larger tests, convert bytes to KB or MB to make more sensible numbers.

(8192000000 / 1024 / 1024) / ((2 * 60) + 41.618)

Bytes converted to MB / (2 minutes + 41.618 seconds)

This gives us an average of 48.34 megabytes per second over the duration of the test.

Read

We can also use dd to test the read speed of a disk by reading the file we created and timing the process. Before we do that, we need to flush the file cache by writing another file which is about the size of the RAM installed on the test system. If we don’t do this, the file we just created will be partially in RAM and therefore the read test will not be completely read from disk.

Create a file using dd which is about the same size as the RAM installed on the system. The below assumes 2GB of RAM is installed. You can check how much RAM is installed with free.

dd if=/dev/zero of=/mnt/mount1/clearcache.tmp bs=4k count=524288

Now for the read test of our original file.

time sh -c "dd if=/mnt/mount1/test.tmp of=/dev/null bs=4k"

And process the time result the same was as when writing.

Bonnie++

Bonnie++ is a small utility with the purpose of benchmarking file system IO performance. It’s commonly available in Linux repositories or available from source from the home page.

On Debian/ Ubuntu based systems, use the apt-get command.

apt-get install bonnie++

Just like with DD, we need to minimise the effect of file caching and therefore the tests should be performed on datasets larger than the amount of RAM you have on the test system. Some people suggest that you should use datasets up to 20 times the amount of RAM, others suggest twice the amount of RAM. Whichever you use, always use the same dataset size for all tests performed to ensure the results are comparable.

There are many commands which can be used with bonnie++, too many to cover here so let’s look at some of the common ones.

  • -d – is used to specify the file system directory to use to benchmark.
  • -u – is used to run a a particular user. This is best used if you run the program as root. This is the UID or the name.
  • -g – is used to run as a particular group. This is the GID or the name.
  • -r – is used to specify the amount of RAM in MB the system has installed. This is total RAM, and not free RAM. Use free -m to find out how much RAM is on your system.
  • -b – removes write buffering and performs a sync at the end of each bonnie++ operation.
  • -s – specifies the dataset size to use for the IO test in MB.
  • -n – is the number of files to use for the create files test.
  • -m – this adds a label to the output so that you can understand what the test was at a later date.
  • -x – is used to repeat the tests n times. Change n to the number of how many times to run the tests.

bonnie++ performs multiple tests, depending on the arguments used, and does not display much until the tests are complete. When the tests complete, two outputs are visible. The bottom line is not readable (unless you really know what you are doing) however above that is a table based output of the results of the tests performed.

Let’s start with a basic test, telling bonnie++ where to test and how much RAM is installed, 2GB in this example. bonnie++ will then use a dataset twice the size of the RAM for tests. As I am running as root, I am specifying a user name.

bonnie++ -d /tmp -r 2048 -u james

bonnie++ will take a few minutes, depending on the speed of your disks and return with something similar to the output below.

Using uid:1000, gid:1000.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ubuntu 4G 786 99 17094 3 15431 3 4662 91 37881 4 548.4 17
Latency 16569us 15704ms 2485ms 51815us 491ms 261ms
Version 1.96 ------Sequential Create------ --------Random Create--------
ubuntu -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
 files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
 16 142 0 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 291us 400us 710us 382us 42us 787us
1.96,1.96,ubuntu,1,1378913658,4G,,786,99,17094,3,15431,3,4662,91,37881,4,548.4,17,16,,,,,142,0,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,16569us,15704ms,2485ms,51815us,491ms,261ms,291us,400us,710us,382us,42us,787us

The output shows quite a few statistics, but it’s actually quite straight forward once you understand the format. First, discard the bottom line (or three lines in the above output) as this is the results separated by a comma. Some scripts and graphing applications understand these results but it’s not so easy for humans. The top few lines are just the tests which bonnie++ performs and again, can be discarded.

Of cause, all the output of bonnie++ is useful in some context however we are just going to concentrate on random read/ write, reading a block and writing a block. This boils down to this section:

Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ubuntu 4G 786 99 17094 3 15431 3 4662 91 37881 4 548.4 17
Latency 16569us 15704ms 2485ms 51815us 491ms 261ms

The above output is not the easiest output to understand due to the character spacing but you should be able to follow it, just. The below points are what we are interested in, for this example, and should give you a basic understanding of what to look for and why.

  • ubuntu is the machine name. If you specified -m some_test_info this would change to some_test_info.
  • 4GB is the total size of the dataset. As we didn’t specify -s, a default of RAM x 2 is used.
  • 17094 shows the speed in KB/s which the dataset was written. This, and the next three points are all sequential reads – that is reading more than one data block.
  • 15431 is the speed at which a file is read and then written and flushed to the disk.
  • 37881 is the speed the dataset is read.
  • 548.4 shows the number of blocks which bonnie++ can seek to per second.
  • Latency number correspond with the above operations – this is the full round-trip time it takes for bonnie++ to perform the operations.

Anything showing multiple +++ is because the test could not be ran with reasonable assurance on the results because they completed too quickly. Increase -n to use more files in the operation and see the results.

bonnie++ can do much more and, even out of the box, show much more but this will give you some basic figures to understand and compare. Remember, always perform tests on datasets larger than the RAM you have installed, multiple times over the day, to reduce the chance of other processes interfering with the results.


13 Comments

Leon

20-May-2014 at 12:07 pm

You should use ‘oflag=direct’ with dd, bypassing system buffers to create a more reliable benchmark result.

Also, bonnie++ ships with the bon_csv2html utility. If you cat the entire output produced by bonnie++ and pipe it through bon_csv2html, it’ll produce a nice html page for you.

    james.coyle

    20-May-2014 at 1:30 pm

    It’s true but I wouldn’t recommend the HTML generation as generally you (or I at least) would use this on a headless machine and getting the HTML file into a browser can be tedious. I’d recommend learning the bonnie++ output from day one and after you’ve seen it a few times it’ll be like looking at the Matrix.

oblique

2-Jul-2014 at 4:21 pm

Nice article straight to the point and a succint illustration.

I know you didn’t intend to make the two tests directly comparable but the difference between the dd and bonnie write rates (48MB/S vs 17MB/S) is interestingly huge. Not sure what bonnie does with intelligent writing (vs dd 4k blocks), but from the bonnie output looks like its test was CPU bound?

    james.coyle

    3-Jul-2014 at 8:11 am

    Hi Oblique,

    Thanks for the comment.

    Bonnie++ performs multiple tests to calculate the benchmark result – I didn’t dive deep enough under the covers to understand exactly what it’s doing so I can’t really explain the difference.

    If I get time I’ll run some tests and try and figure out why there is such a major difference and post the results.

ddd

23-Nov-2014 at 10:25 pm

1) Are you doing the test on SSD? 50MB/s for 4KB sequential write is pretty good. I was thinking something was wrong with my storage setup that I only got 10MB/s with HDD

2) Why reinvent the wheel with the time command when dd gives the speed already?

8192000000 bytes transferred in 159.062003 secs (51501929 bytes/sec)

3) when you use dd for read test, why are you writing it back to the same file instead of just discard the data (of=/dev/null)?

It is an interesting topic but some points in the article seems misleading.

    james.coyle

    24-Nov-2014 at 5:01 pm

    Hi ddd,

    1) Yes, these tests are done on SSD storage.
    2) Because that doesn’t include the final flush to disk – that’s why the times/ speeds are different.
    3) You are correct, in fact you don’t need to include the ‘of’ argument at all – I’ve updated the post.

Joshua Grigonis

17-Dec-2014 at 12:34 am

I found that when doing the read tests using dd, if we didn’t specify an output file, we got terrible results.

So, when I did:

time sh -c “dd if=/mnt/mount1/test.tmp bs=4k”

I only got around 35 MB/s

However, when I did:

time sh -c “dd if=/mnt/mount1/test.tmp of=/dev/null bs=4k”

I got the more expected 274 MB/s

I figured out what was going on by piping the original command to wc, and seeing that it was actually writing through stdout.

    Sven

    21-Mar-2015 at 2:01 pm

    same here, leaving out the of=/dev/zero gave me read speed of something like 2M/sec

    The One

    20-Oct-2015 at 10:01 pm

    That’s because DD without “-if” is pumping everything to STDOUT.
    If the file wasn’t just null bytes, you’d be seeing a lot of crap on the console :-)

The One

20-Oct-2015 at 10:09 pm

To benchmark read speed I would recommend bypassing the file cache:

dd iflag=direct if=yourFile of=/dev/null bs=1M

venkata

21-Jun-2016 at 10:42 pm

When do I use a -y option to run the Bonnie++ and what advantage does it give?

Andre Smit

13-Jan-2017 at 8:42 am

Hello

You can also mount filesystems with -o sync or put sync as an option in /etc/fstab, and mount -o remount the same filesystem.

This removes disables buffering, and then yout sync write will show true hardware speed. However, most modern drives will need something like bs=4096 or greater to produce a realistic speed. As files written to the disk from buffer once you remove sync option will be written in chunks and not sector by sector.

Dan

7-May-2021 at 10:59 am

iotop can be fun to watch while you run these tests.

Leave a Reply

Visit our advertisers

Quick Poll

Do you use ZFS on Linux?

Visit our advertisers