Data Centers

Find detail storage measurement for virtual machines: IOPs

Virtualization requires everyone to become a bit of a storage expert. But what do we look for? IOPs is the key, Rick Vanover explains.

When we first started virtualizing, we looked at a number of different things initially to monitor performance: CPU usage, memory overhead, network saturation, and free space on storage. I like to equate those to the warning lights of the cars we drive, as they really are only indicators of serious problems (like a check engine light).

In particular, storage is one area where you will potentially spend a lot of time monitoring, and hopefully less time troubleshooting. I like to take a three-phased approach to storage, in general. This approach is metric-driven, and then impacts every other area of the storage practice for virtualization.

Historically, I only monitored free space, the check engine light of storage as it were. I then advanced to monitoring latency (multiple types of latency) on datastores, which is a good indicator of a volume’s performance. Nowadays, I’ve zoomed into looking at IOPs for virtual machines. This is a good way to see what is really going on.

But, it’s important to note that all of these measures are important. Others are important as well, but generally speaking free space is my generic measurement, latency is the medium-measurement, and IOPs is the detail measurement. And the latter applied to a specific virtual machine is excellent visibility.

IOPs are simply Input/Output operations per second, and are nothing new in terms of measuring storage systems. While I hope to avoid having the metric fetish phenomena (over-measuring), it is important to have visibility into this specific value through the abstraction of virtualization.

There are a lot of ways to obtain the IOPs and other storage values. The vSphere Client, storage solutions, and monitoring tools can all report these values. One way to do this is with storage solutions, and in Figure A below, a VM running on a Tintri datastore is reporting its IOPs in a VMware virtual environment:

Figure A

Click to enlarge.

In this example, the virtual machine leading this view ending in 007 is consuming 1,643 IOPs. This particular storage system is a hybrid rotational storage system with high performance non-rotational storage resources. The Tintri moves the hot spots of a VMDK to the non-rotational storage. Consider that the average SATA drive may push less than 100 IOPs; the one VM leading this view at a quiet time could require 16 drives to satisfy over 1,600 IOPs just by itself. My next task is to figure out what virtual machine 007 is doing! This is the visibility we need for our storage in virtualized environments.

There are a lot of ways to get this visibility into your virtual machines -- and use the right metrics (free space, latency and IOPs). What ways do you measure your virtual machine storage in detail? Share your comments below.

About

Rick Vanover is a software strategy specialist for Veeam Software, based in Columbus, Ohio. Rick has years of IT experience and focuses on virtualization, Windows-based server administration, and system hardware.

3 comments
zmx
zmx

Hi Rick! It depends on what should be the goal of measurement. IOPs is only one of the parameters that determine the quality of IO subsystem. The total amount of IOPs depends on the storage used, on the physical interfaces of the storage/host etc. More interesting, according to me, is the number of IO operations that are waiting for processing. When the queue is small or empty, the IO is (from point of view of the virtual machine) okay. The problem usually starts when the queue size will increase. Then it is important to measure the IOPs, but also the amount of data transmitted. Regards Zdenek

lkarnis
lkarnis

Rick: Not everyone has a Tintri SAN. You can measure the same thing in the vSphere Client: - Log in to ESXi host (or vCenter and point at ESXi host) - Performance tab > Chart Options - Select Datastore > Real Time - Uncheck all Objects except the datastore you want - In Counters check 'Average Read Requests per Second' and 'Average Write Requests per Second' (sadly there is no I/O Operations per Second) - Click OK and check out the latest/maximum/minimum values. You can also set up I/O workloads with IOMeter (iometer.org)

wdewey@cityofsalem.net
wdewey@cityofsalem.net

You give stats for how many SATA drives it would take to satisfy the IOPS, but aren't SAS drives more common in a SAN? SATA would be better for a system that has lower IOPS needs, but higher storage density needs such as an archival system. For a SAS system to support VM 007 you would only need 6 SAS drives.