Input/output operations per second (IOPS, pronounced eye-ops) is a performance

measurement used to characterize computer storage devices like hard disk drives
(HDD), solid state drives (SSD), and storage area networks (SAN).

What is the difference between IOPS and throughput?

Storage performance: IOPS, latency and throughput

IOPS, latency and throughput and why it is important when troubleshooting

storage performance

In this post I will define some common terms regarding storage performance. Later we
will see different tools for stressing and measuring this. With storage I mean all forms
of mechanical disk drives, it could local SATA drives in PCs, it could also be SCSI disks in
servers or large amounts of RAID connected expensive drives in a SAN.

The most common value from a disk manufacturer is how much throughput a certain
disk can deliver. This number is usually expressed in Megabytes / Second (MB/s) and
it is easy to belive that this would be the most important factor to look at. The
maximum throughput for a disk could be for example 140 MB/s, however for several
reasons this is often not as critical as it seems and we shall not expect to actually
achieve that amount of throughput on a typical disk in production. We shall return to
this value later.

Next term which is very common is called IOPS. This means IO operations per
second, which means the amount of read or write operations that could be done in one
seconds time. A certain amount of IO operations will also give a certain throughput of
Megabytes each second, so these two are related. A third factor is however involved:
the size of each IO request. Depending on the operating system and the
application/service that needs disk access it will issue a request to read or write a
certain amount of data at the same time. This is called the IO size and could be for
example 4 KB, 8 KB, 32 KB and so on. The minimum amount of data to read/write is the
size of one sector, which is 512 byte only.

This means:

Average IO size x IOPS = Throughput in MB/s

Each IO request will take some time to complete, this is called the average latency.
This latency is measured in milliseconds (ms) and should be as low as possible. There
are several factors that would affect this time. Many of them are physical limits due to
the mechanical constructs of the traditional hard disk.
The disk has plates that rotates with a speed expressed in Revolutions Per Minute or
RPM. That is the number of times the plates will do a full rotate in one minutes time.
Since the disk arm and the head (who does to actual read or write) is fixed in one
position it will often have to wait for the plate to spin to the right position. Common
RPMs is 5400 or 7200 for consumer SATA disks and 10000 or 15000 RPM for high
performance server / SAN disks.

This gives:

So for the disk to spin the plate one full rotation takes from 4 to 11 milliseconds
depending on the RPM. This is called the Rotational Delay and is important since the
disk can at any moment be given an instruction to read at any sector of any track. The
disk spins at all times and it is most likely that the correct sector will not (by pure luck)
be right under the disks read head, but instead the head will have to literally wait for
the plate to spin around for the wanted sector(s) to become reachable.
Next factor for the latency time is movement of the disk arm and head itself, while
waiting for the sector to come spinning it would also have to position itself at the exact
right track to be able to catch the information while it comes flying by. The arm is
fixed at one end, but can swing from the inner to the outer part of the disk area (see
above) and by so it can reach any position of the disk, even if it has to sometimes wait
for the correct area to then spin into its scope. The time it takes to physically move the
head is called the seek time. (When looking at the specification of a disk you could see
the average seek time, the lower amount of seek time the faster is the movement of the
arm.)

Once the arm is in the right position and the moment the plate has rotated enough we
can begin to read something. Now depending on the requested IO size this will take
different amounts of time. If the IO size was very small (minimum is 512 byte) then the
IO is completed after the first sector is read, but if the request was 4 KB or 32 KB or
even 128 KB then it would take longer. This is called the transfer delay, which is the
amount of time it will take to do the read/write. We will also hope that the next data is
located on the next incoming sectors on the same track. Then the arm can wait for more
data to roll in and just continue reading. If the data however is located on different parts
of the disk we would have to re-position the arm and wait again for the disk to spin.
This is why fragmentation on a file system is so hurtful for performance. The
more data that is continuously placed on the disk the better.

(An interesting note is that you can actually hear the arm move, that is: the typical
intensive clicking noise from a hard disk is the movement of the disk arm. When doing
sequential access the disk is very silent.)

There is a relation between the IO size and the IOPS, as in if the IO size is small we could
get higher amounts of IOPS and reach a certain amounts of throughput (MB/s). An
example from a SATA disk in my computer when running a disk stress tool:

IO size = 4 KB gives IOPS = 29600 (And then IO size x IOPS = Throughput )

4096 x 29600 = 121 MB/s
When doing a test with larger IO requests, 32 KB, the amounts of IOPS drops:

IO size = 32 KB gives IOPS = 3700

32784 x 3700 = 121 MB/s (IO size x IOPS)

So larger IO request sizes could mean less IOPS but still the same amount of
throughput.

This leads to the final factor: how much of the requested data is actually located near
each other? The term for this is if the data is accessed sequential or random. With
random we mean that not many IO requests are for continuos data, but instead many
small requests located on different parts of the disks. With random access we will get
the highest pressure on the disk, because we would have to do the actions that cost
time very often that is: moving the disk arm to a new position and wait for the disk to
spin around. Then do some small read or write operation, perhaps 4 KB, and then move
on to a new place.

Random access will drop the amounts of IOPS extremely and also the throughput. On
the SATA disk I used as example above, which we saw could give us 29600 IOPS of 4 KB
= 121 MB/s. High values, but these numbers were for sequential access. If doing the
same test but only doing random access we get this result:

IO size (random) x IOPS = Throughput

4 KB x 245 = 0,96 MB/s

So the same disk, but with another access pattern and the performance will drop
sharply. The IOPS went from 29000 to 245 and the throughput decreased from 121 MB/s
to less than 1 MB/s and this is why throughput is not the most important value
to study. It would not matter if this disk was capable of delivering, say, 500 MB/s if the
typical access is random, as we would still get just a few MB/second.

Since most of actual disk access from most production systems is small IOs randomly
spread over the disk the most critical factors will be to get as many IOPS as possible,
with low average latency. This would mean that for a single disk we will want to
minimize the rotational delay by having as high RPM as we can (preferably 15K RPM for
important servers) and minimize the seek delay when the disk head is moving into
position by selecting disks with low average seek time.

As we see that when doing random access IOs with the number of IOPS down to a few
hundreds the next natural step will be to add more read heads, that is: add more
physical disks (often called spindles) that has its own moving components and group
these into RAID systems to get more work done simultaneously.
Or we could look at storage devices without any moving parts, also known as Solid
State Drives (SSD).

http://rickardnobel.se/storage-performance-iops-latency-throughput/