Run vscsiStats for vSphere to see storage activity

vExpert Rick Vanover walks through vscsiStats for vSphere and explains why you'll want to run the storage performance troubleshooting tool before there's an issue to investigate.

When it comes to troubleshooting virtual machine performance, I find that storage and memory are the two resources that frequently deal with the most contention. This isn't necessarily indicative of a problem; however, it's a reminder that we need to be able to effectively troubleshoot these areas.

For storage resources on vSphere environments, one of the best tools to use for troubleshooting is vscsiStats. The tool is built in to the ESXi (it was available in the ESX 3.5 era as a downloadable file).

Big picture notes

Before running vscsiStats, there are two things you need to consider:

  • It is not a real-time display of storage usage (like esxtop).
  • It runs through the tech support mode console, which might need to be enabled on default ESXi installations within the Security Profile configuration. This may be a situation where SSH is a more practical way to run this tool. Virtual console redirection complicates the ability to see the entire screen, but keep in mind the security implications of SSH on the ESXi host. It would be best done on demand only and disabled through the vSphere Client when this task is complete.

Detailed notes

vscsiStats is also used to return one virtual machine's I/O profile for a window of time (usually 30 minutes or so). To get a profile made up, you need to identify the virtual machine. vscsiStats will use a worldGroupID for a virtual machine to use for its collections. Browse in the local tech support mode console to the /usr/lib/vmware/bin/ path. To list the running virtual machines and display the worldGroupID's, run the vscsiStats -l command (Figure A). Figure A

Click the image to enlarge.

The virtual machine I highlighted in Figure A, VBAR1, is the one I want to create an I/O profile on. The worldGroupID on this VM is 2227645, and will be used on the next step.

To run vscsiStats collection for a particular virtual machine, you run the command with the WorldGroupID specified. The full syntax is vscsiStats -t -s -w 2227645 (Figure B). Figure B

Click the image to enlarge.
The collection will run for approximately 30 minutes. Once it is complete, there are a number of commands you can use to view the statistics. One all-encompassing view is the printhistos command, which shows all of the results. This command is: vscsiStats -p all. The more manageable option is to export these values to a .CSV file. The command is vscsiStats -p all -c > tracefile.csv (Figure C). Figure C

Click the image to enlarge.

The export file can be easily moved off of the ESXi host through FTP or placed on a VMFS datastore where the datastore browser on the vSphere Client can access it. This command will move the file to a VMFS datastore:

mv tracefile.csv /vmfs/volumes/DAS-ESXi-RWVDEV-INTRA/

"DAS-ESXi-RWVDEV-INTRA" is the VMFS datastore name for the local disk on host esxi.rwvdev.intra in this example. The VMFS volumes are logically represented in the /vmfs/volumes path of the ESXi host.

With this data, you can interpret a number of things -- most importantly, what is normal for the virtual machine. vscsiStats is the answer to the application vendor who doesn't have a satisfactory answer on the application's I/O behavior. From the tracefile.csv that was created in the previous steps, this can be opened in Excel, and a number of graphs and analysis can be performed.

One of the first categories in the tracefile that has all of the counters, the I/O command length, is recorded. In this example, most of the I/O commands are a 4096 Kb length, which is good. By comparison, if the majority of I/O operations were the 512 Kb lengths, it would take the virtual machine longer to complete everything because it is working with small tasks. This is shown in Figure D. Figure D

Click the image to enlarge.

As a safety measure, make sure you stop all collections when you're finished running vscsiStats. You can do that by entering the vscsiStats -x command.

vscsiStats takes some getting used to, but it's worth it -- running a profile on a virtual machine in a normal running state is a good idea. This way, when it's time to troubleshoot, there will be something to compare it to, and you'll be familiar with the tool.

How do you use vscsiStats? Which metrics are your favorites? Share your comments with the TechRepublic community.