iorate(1)iorate(1)NAMEiorate - tests I/O subsystems under various loads
SYNOPSISiorate [ -a ] [ -l ] [ -n ] [ -s ] [ -u ] [ -v ] [ -d <debug_level> ] [
-c skew_shift ] [ -r target_rate ] [ -f device_file ] [ -p pattern_file
] [ -t test_file ] [ -i iops_file ] [ -o output_base ]
DESCRIPTION
The iorate utility conducts a series of I/O tests. Each test is
defined in test_file using a collection of patterns. Each pattern is
defined in pattern_file. The tests are run simultaneously against all
of the devices in device_file. A useful user guide can be downloaded
from the web page at: http://iorate.org.
Each pattern defines a specific I/O pattern that will be run. Each
pattern can be either read or write, either random or sequential. For
sequential patterns, a maximum number of contiguous sequential tests
can be specified, after which the next random sequence will start at a
new, random location. All I/O for a pattern is of a fixed size.
Each test uses one or more patterns, specified as a percentage of the
I/Os to be performed. The tests specify how long they should run,
where they start in the file, and the size of the device that they
should use for testing. The can also specify how much of the test
period should be ignored for performance characteristic reasons to
allow time for the test to achieve a 'steady state,' which can be use‐
ful when testing disk arrays with large amounts of cache.
Iorate generates a log file of the test being run. The log file con‐
tains a copy of the configuration files being used (device_file, pat‐
tern_file and test_file). These will not have any comments, and will
be printed directly from the information that will be used to drive the
actual tests. These copies, if split out into the separate input
files, will reproduce the same test. They are included in the log to
show exactly what was run, and to allow that test to be reproduced at
any later time. The log also records the starting and ending times of
each test, by device. Any errors encountered are also included in the
log.
Note that if an iops_file was used, or if the target rate was scaled
using a target_rate, then the iops target for each test in the
test_file section of the log will have a value reflecting these target
rate adjustments. Spliting these sections out to run a new test will
run the exact same test again, at the adjusted target rate, without the
need for target iops or rate adjustments. If the adjustments (iops
file and/or target rate percentage) are applied again, the tests may be
altered.
In addition to the log file, iorate generates a performance file for
the test group. The performance file contains detailed statistics on
each test run on each device. Each line in the file represents the
data for a single test on a single device. The first line of the file
is a header that describes the contents of each test line. The file
uses tab separated values, that are easy to read with scripts or
spreadsheet programs (though the alignments are a challenge if read
with a standard editor or simply printed). The fields are:
test_number -- Number of the test being run
test_name -- Name of the test being run
device_num -- Number of the device for this line
device_name -- Name of the device for this line
total_sec -- Total seconds the test was run
measured_sec -- Time that results are measured over
reads -- Number of reads from this device
KB_read -- KB of data reads from the device
read_resp -- Avg. response time for all device reads
writes -- Number of writes to this device
KB_written -- KB of data written to the device
write_resp -- Avg. response time for all device writes
total_I/Os -- Number of I/Os to this device
total_KB -- KB of data transfered to the device
total_resp -- Avg. response time for all device I/Os
I/Os_per_sec -- Avg. number of device I/Os per second
KB_per_sec -- Avg. KB of data transferred per second
dev_copy -- Which copy of the device this is for (for devices with
multiples)
Each line is a set of performance numbers for a single device. To get
the total for a given test, the numbers for each of the devices needs
to be summarized for each test. The gen_sums AWK program provided will
generate such test summaries, which are often more useful for compar‐
isons.
If the pair of output files (performance and log) are available from
any test, the numbers can be examined, and the test can be reproduced.
USAGE
The log file is usually named iorate.log, and the performance data is
in iorate.perf. If the -o option is used, log file will be out‐
put_base.log and the performance data will be output_base.perf. If
large numbers of test passes will be run, it is suggested that sub-
directories be built for each group of test. The output_base parameter
can include sub-directory names as well, but those sub-directories must
exist before iorate begins (iorate will not create sub-directories).
The -l limits option causes iorate to display file limits and exit (no
testing). This is used in the makefile to verify support for large
file sizes.
The -n no testing option turns off test execution. The program will
still do all setup checks, including a seek to test the size of each
file. The program will still fork off a child process to do the test‐
ing. However, the child will exit quickly.
The -s silent option stops all but the error and warning output to
standard out. The other information normally reported, such as test
configurations and progress, is still recorded in the log file for
later reference.
The -u use direct I/O option sets the O_DIRECT (or appropriate) flag
when opening the data files for testing. This should cause file system
buffers to be bypassed and provide a more accurate measure of the true
array performance.
The -a allow causes the checks on read sizes to be lenient. This
option is helpful on some platforms where the reads from files on file
systems may not return as many bytes as the read I/O is requesting.
The -v verbose option turns on verbose mode. Since this will generate
log information for every I/O of every device, it is not recommended
for use in a lengthy test. It will have a dramatic effect on the I/O
rates of many tests, and should not be turned on while real test num‐
bers are to be generated. Note that multiple processes (one per active
test process) will be writing to the same log file, so the output lines
may get a bit messy with more than one active device. The log file
will be very busy, and may grow to be quite large.
The -d <debug_level> debug option tells iorate to turn on a number of
debugging outputs. This will provide a large number of messages that
may not be understandable to someone not reading the program source
code. The optional debug_level allows the user to shift from some
information (10) to an excrutiatingly detailed level of information
(1000+).
The -c skew_shift option tells iorate to alter the sub-LUN skew data.
This shifts skew_shift percent of the data between the target levels,
with hot regions becoming cold, cold regions becoming warm, and warm
regions ecoming hot. This is limited to be 0 and 99. Note that the 99%
shift hsd 100% of the data on a new performance level, and thus is not
close to the 0% level.
The -r target_rate option tells iorate to run at a reduced target rate.
This will cause each test to run at this percentage of the maximum iops
given for that test (this has no effect if an iops has not been speci‐
fied for that test). The target (base) iops rate for each test may be
specified in the test_file, or in the iops_file. Valid target rates
are from 0 (no target rate used - leave iops values alone) to 100%
(full test).
PARAMETER FILES
Iorate requires that all three configuration files be properly format‐
ted. Any errors in the files will be reported as the program starts,
and may cause testing to be aborted. In general, anything following a
hash mark (#) is considered a comment to the end of the physical line
(marked by a carriage return). Each logical line describes a single
item (device, pattern, test, or iops rating) and ends in a semicolon
(;). Logical lines (single items) can span many input lines.
In general, each item will have an item type identifier (device, pat‐
tern, test, or target) and then information about that item. The
details of the various options for each item type are noted in the file
descriptions below.
When providing sizes, a number can be followed by a sizing factor (KB,
MB, GB, TB). Each step will increase the size by a multiplier of 1024
(not 1000). When providing time values, a number must be followed by a
time type (sec, min, hour) which will scale the number to seconds.
When providing numbers, general math functions also work, so 4 * 4 GB
is the same as 16 GB.
DEVICE FILE
Iorate will assume a device_file name of devices.ior if none is speci‐
fied. Each device entry describes a single device, such as:
Device 1 = "/mnt/datafile.tst" capacity 3GB;
Any valid file name can be given, whether a raw device file or the name
of a file system file that exists and has the stated size. Device num‐
bers must be unique, but are also optional (if not provided, device
numbers will be assigned in order automatically - a record of which is
provided in the log file). The capacity is the usable space in the
device for testing - the actual size can be much larger.
There is a minumum offset of 8 KB, which is also the default, to make
sure that any device header such as a label is not overwritten. Alter‐
nate values can be set with offset <size>. The capacity of the device
is the usable capacity starting at the offset point. I/O can be tar‐
getted at the last 2 GB of a 36 GB device using offset 34 GB capacity 2
GB.
The minumum block size, which is also the default, is 512 B for most
operating systems (1024 B on HP-UX). Alternate values can be set with
block size <size>. This block size is the minimum I/O size for the
device, and all I/O will be aligned to this size, as well as being a
multiple of it. The patterns in use from any test must be a multiple
of this size.
Rather than placing the same file information in the device file repet‐
itively, a single definition can have multiple tasks set to access it
using count <count>. Each process will open the file separately,
choosing a random start position and performing seeks and I/O sepa‐
rately from the others.
The file can be flagged as being read-only by specifiying read only.
This will cause iorate to open the file for reads but not writes - and
any tests attempting to do writes will cause iorate to fail. Unless
the file is set to read only, iorate will attempt to open the file for
reads and writes - even if none of the active patterns in the tests
will do writes.
There is also an option to lock the file by specifiying lock. This
will use the fcntl() system call to lock the area of the device speci‐
fied by the offset and capacity. Note that these are exclusive (for
writes), non-blocking locks - if they cannot be obtained, the entire
test will be aborted. Note that multiple read locks on the same area
of a file are permitted, so setting the devices to read only will allow
them to be shared even if locking is requested. If the device is not
set to read only, then iorate will attempt to aquire a write lock
(exclusive) on the defined region of the file - even if none of the
active patterns in the tests will do writes.
PATTERN FILE
Iorate will assume a pattern_file name of patterns.ior if none is spec‐
ified. Each pattern entry describes a single pattern, such as:
Pattern 1 = "2K Seq Read" io size 2KB sequential read;
The pattern number must be given, since that is how the patterns are
chosen from the tests. They need not be sequential.
The io size <size> parameter specifies the size of all I/Os that will
be driven from this pattern. To mix I/O sizes, tests can be configured
that use multiple patterns.
The area to run the pattern against can be specified with from <size>
or from <count>% to set the starting location, and size <size> or size
<count>% to set the size to test against (starting at the from loca‐
tion). Any percentage sizes are scaled from the active size of the
test using this pattern.
By default, the pattern will do all sequential read. Setting random
will change the pattern to random address selections. For a mix, use
max sequential <count> to limit the number of sequential I/Os that will
be done in a row before a new, random location is chosen. Note that
setting <count> to 10 will allways to 10 sequential I/Os before select‐
ing a new location. To mix the I/O types, select write for 100% write
activities, or read <count>% to have the pattern do a mix of reads and
writes.
To more closely model a real application, I/Os to the same address can
be repeated. Setting reuse <count>% will cause that percentage of I/Os
to be done to the same address as another recent I/O. Setting history
<count> will determine how long of a history of I/Os is kept to reuse.
Further modeling is available by creating 'locality of reference'
zones. By setting zone <size> as a fixed size, or zone <count>% as a
percentage of the pattern test area, iorate will create count <count>
localities to do I/Os against. To model 'moving hot spots,' specify
limit <count> to determine how many I/Os will be done to a given local‐
ity before a new one is chosen. Defining any of zone, count, or limit
will cause default values to be set for the others if none are speci‐
fied.
TEST FILE
Iorate will assume a test_file name of tests.ior if none is specified.
WARNING: the test-fx.ior tests are destructive to all files/devices
used for testing. Files/devices that have data contained in them should
be included in the device file only if extreme care has been taken to
ensure that all test patterns are read-only. Because of this danger,
the default test file uses read-only patterns. Even so, great care
should be taken whenever any entry in the devices file has data that is
valued. Failure to do so has resulted not only in lost data, but in
one case caused such damage to the root file system (random writes of
garbage data) that the server had to be reinstalled from tape. There
is no attempt made to check for other users of any device listed, the
tests are simply run.
Each test entry describes a single test, such as:
Test 1 = "Mixed 2K Seq" for 120 sec ignore 20 sec 50% pat 1, 50%
pat 2;
Test numbers must be unique, but are also optional (if not provided,
device numbers will be assigned in order automatically - a record of
which is provided in the log file). The duration of the test is set
with for <time>. To make sure that the performance during any start-up
period is not measured, but rather the steady-state numbers, setting
ignore <time> causes the performance statistics to be reset that far
into the test time. In the event that one test may leave work in for
the test storage, such as writes sitting in the cache of a disk array,
pause <time> will introduce that length of delay before the test is
activated.
The area to run the test against can be specified with from <size> or
from <count>% to set the starting location, and size <size> or size
<count>% to set the size to test against (starting at the from loca‐
tion). Sizing that are based on percentages are based on the capacity
of each device being tested - so if there are devices with different
capacities, the tests will resize themselves for each one.
To limit the level of I/O generated by this test, set <count> iops.
This sets an I/Os per second target for the entire test - and since
there are separate processes run for each active device defined, they
will each target their share independently. This means that if there
are 10 active devices, and 100 iops is specified, then the program will
attempt to drive 10 iops to each device. In the event that one device
cannot do its share, the other devices will not be tasked will addi‐
tional work, as the test process working on each devie are not aware of
each other.
Finally, a list of patterns to run is included. Each pattern is speci‐
fied as <count>% pat <pattern>, where each percentage is a whole number
(decimals ignored), and the total percentages of all patterns must
equal exactly 100%.
IOPS FILE
Iorate assumes that no iops_file is available. Unless an iops_file is
given on the command line or through the environment variable, the only
iops limits will be set in the test file itself. If there is no iops
limit specified for a test, then I/O will be generated as fast as pos‐
sible on every device. When a file is given, each entry sets a new
iops limit for a given test, such as:
Target test 5 at 2000 iops;
This is most useful for times when a given set of tests is to be run on
differing configurations, and it is not known ahead of time what the
upper limit of each test will be (maximum test rate). From there, it
may be interesting to run the same tests at some percentage of the max‐
imum rate. If a full speed pass of a test has already been run, then
the gen_iops AWK program provided will generate such an iops_file that
matches the performance seen it that run. Together with a new tar‐
get_rate, a pattern can be tested for response times at various per‐
centages of the maximum attained throughput for each test.
ENVIRONMENT
The command line file names can be provided for in the environment,
with:
IOR_DEV_FILE
The name of the device_file to use.
IOR_PAT_FILE
The name of the pattern_file to use.
IOR_TEST_FILE
The name of the test_file to use.
IOR_IOPS_FILE
The name of the iops_file to use.
IOR_OUTPUT_BASE
The base name of the output files.
AUTHOR
Written by Vince Westin (vince.westin@emc.com), with a lot of assis‐
tance from the EMC Engineering Team. Copyright by EMC Corporation,
1997-2011. All rights reserved. The latest version is available from
the web page at http://iorate.org. Use of the latest version is
strongly encouraged. Check the README and README.versions files from
the latest version for new features and bugs from earlier revisions
that have been fixed.
This code is the property of EMC Corporation. However, it may be used,
reproduced, and passed on to others as long as the contents, including
all copyright notices, remain intact. Modifications to, or modified
versions of these files, may also be distributed, provided that they
are clearly labeled as having been modified from the original. In the
event that modified files are created, the original files are to be
included with every distribution of those modified files. Inclusion of
this code into a commercial product by any company other than EMC is
prohibited without prior written consent.
Having said the legal stuff, this code is designed to provide a good,
generic tool for testing I/O subsystems under various kinds of loads.
If you have suggestions for improvements in this tool, please send them
along to the above address.
SEE ALSOread(2), write(2), seek(2), open(2), close(2), fcntl(2)
October 2011 iorate(1)
