Data Centers

NVMe: A cheat sheet

This primer about NVM Express covers the variety of form factors that NVMe drives come in, as well as how businesses can benefit from the storage technology.

As adoption of solid-state drives (SSDs) increases in enterprise, professional, and consumer markets, the practice of shimming flash memory onto interfaces and form factors designed for traditional mechanical hard drives for the sake of cross-compatibility is falling out of style. In order to fully utilize the potential of solid-state storage technology, industry stakeholders have devised NVM Express as a standard interface for SSDs to communicate with computers.

TechRepublic's cheat sheet to NVM Express is a quick introduction to the high performance, flash-first storage interface specification. This guide will be updated periodically as new storage technologies that utilize the interface are released.

SEE: Quick glossary: Storage (Tech Pro Research)

What is NVM Express?

NVM Express (NVMe), or the Non-Volatile Memory Host Controller Interface Specification, is a device interface specification that was devised to take advantage of the characteristic advantages of flash-based storage technologies such as NAND and 3D XPoint, marketed as Optane by Intel.

Prior to NVM Express, SSDs connected to computers via Serial ATA or SAS, and communicated using Advanced Host Controller Interface (AHCI). These technologies were designed for traditional mechanical hard drives, which have practical physical limitations in random access of data, and the speed with which data can be transferred at once. By late 2010, mainstream consumer SSDs were being bottlenecked by the comparatively limited throughput granted by SATA. In the professional and prosumer market, this limitation was overcome by creating SSDs that connected via PCI Express.

This strategy only allowed manufacturers to overcome the 6 Gb/s (750 MB/s) speed limitation inherent to SATA, with SSDs topping out around 550 MB/sec, when accounting for overhead. However, these drives either used AHCI—retaining almost all the random I/O performance bottlenecks—or used a custom interface specification, with operation varying wildly between vendors and, to an extent, between models of the same vendor. Because of the custom design of these drives, device-specific software drivers were required for use; as a result, early PCI Express SSDs were more complicated to use as a boot drive on Windows, or to use with Linux if the vendor did not support it.

NVM Express eliminates those issues by providing a predictable common interface for drives to use, with a common driver specification eliminating the need for device-specific drivers, taking the guesswork out of using these SSDs.

SEE: All of TechRepublic's cheat sheets and smart person's guides

Of note, traditional hard drives have generally come in 3.5" and 2.5" form factors, and SSDs have typically been available as 2.5" or mSATA cards. NVMe-linked drives have a variety of form factors—the most common is the M.2 format, which is 22mm wide with lengths of 30, 42, 80, or less often 110 millimeters. (M.2 includes SATA capabilities for backward compatibility.) The NF1 format is a data center-targeted variant of M.2 that allows for hot swapping. U.2 drives, formerly known as SFF-8639, are 2.5" drives with a specialty connector to provide PCI Express signaling. Some high-end drives are only available as PCI Express cards, typically half height, half length (HHHL).

Additional resources

Why does NVM Express matter?

NVM Express allows drive manufacturers to create higher-performing drives without being limited by throughput bottlenecks of SATA/SAS and AHCI, which were intended for traditional platter HDDs. When working with multiple files on traditional hard drives, the drive head must move at high speeds across various parts of the drive to read and write data to the disk. While caching and other firmware tricks mask this, traditional HDDs effectively can only work with one file at a time.

As a result, AHCI—which was designed for traditional HDDs—has only one command queue, with a maximum of 32 commands per queue. As this mechanical limitation does not exist in solid-state drives (which lack a drive head), this attribute of AHCI is extremely disadvantageous. NVMe addresses this by allowing up to 65535 command queues, which each can have up to 65536 commands per queue. (This is offered by the specification, though practically taking advantage of this requires the drive controller to support it.)

Likewise, NVMe offers 2048 MSI-X interrupts, while AHCI offers a single interrupt with no steering. Overhead on NVMe is similarly reduced, as 4K command parameters are performed in one fetch, while AHCI requires two host RAM fetches. Additionally, NVMe does not require locking for parallel instructions. Taken as a whole, NVMe has a flash-first philosophy that does not constrain drives from using the solid-state attributes of flash, which in turn leads to higher performance.

Multiple factors, such as the type of flash used, as well as drive controller and firmware, can influence the relative performance of solid-state drives. The most equitable comparison at hand would be between the WDS100T2B0B and the WDS100T2X0C. Both are relatively recent 1 TB SSDs from Western Digital, using 64-layer 3D TLC NAND flash. The former is a SATA/AHCI drive, which is rated for 560 MB/s and 530 MB/s sequential read and write speed, respectively, with 95K and 84K random read and write IOPS. For contrast, the latter is a PCIe/NVMe drive, which is likewise rated for 3400 MB/s and 2800 MB/s sequential read/write, with 500K and 400K random IOPS. This level of performance is only possible by merit of the use of NVM Express.

Additional resources

Who does NVM Express affect?

In the enterprise, adoption of NVM Express can greatly increase database performance. In 2015, researchers from the University of Southern California and San Jose State University working in conjunction with Samsung found that (PDF) architectural differences in NVMe resulted in a "4x decrease in system software overheads," as of version 3.14 of the Linux kernel. Further, the researchers indicate that in a head-to-head comparison of a single NVMe drive compared to a single SATA drive, the NVMe drive "can provide performance benefits of up to 8x" on database workloads. Similarly, a single NVMe drive outperforms a a RAID0 array of four SATA SSDs by "up to 5x."

For professional and prosumer use cases—such as graphics editing and video production—the benefits are slightly more modest, as these activities rely more on raw read/write speeds than random I/O capabilities of drives. That being the case, performance increases of 8x—as seen on databases—are not as realistic for these use cases. Looking back at the speed examples of the two drives compared in the last section, NVMe offers about a 6x increase in read speed, and a 5.25x increase in write speed over a similar SATA SSD.

Additional resources

When was NVM Express released?

The first release of the NVMe specification was in March 2011, while the current version (1.3c) was released in May 2018.

Every major operating system includes support for NVM Express. Microsoft included native support for NVMe in Windows 8.1 and Server 2012 R2 in October 2013, with support backported to Windows 7 and Server 2008 R2 via Windows Update. Apple added support for NVMe in OS X Yosemite 10.10.3 in April 2015. Support for NVMe in Linux was released in kernel version 3.3 in March 2012. This support was added to Chrome OS in February 2015.

Support for NVM Express also exists in OpenBSD 6.0, NetBSD-current, FreeBSD 10.2, DragonFly BSD 4.6, as well as Oracle Solaris 11.2 and the OpenSolaris fork illumos. VMware supports NVMe as of vSphere 6.0. Naturally, successive versions of these products (Windows 10, OS X 10.4 Mojave, etc.) retain support for NVM Express.

Additional resources

How do I get an NVM Express-powered drive?

Users of relatively modern PCs can use NVMe SSDs, with many vendors shipping PCs with NVMe drives integrated. For Mac users, Apple adopted NVMe SSDs starting with the Retina MacBook in 2015, and in the MacBook Pro with Touch Bar in 2016, as well as most configurations of the Mid-2017 Retina iMac and all configurations of the 2017 iMac Pro.

For PCs, newer motherboards often have dedicated M.2 slots for connecting SSDs. For newer motherboards that do not feature these connectors, adapter cards are available that allow M.2 drives to be connected using existing PCI Express slots. Some drive manufacturers offer these bundled with SSDs, though inexpensive generic solutions that work with drives from all manufacturers are commonly available.

Of moderate importance, the performance of NVM Express drives depends on the revision and number of lanes available to it via PCI Express. PCI Express 2.0 supports 500 MB/s per lane, while PCI Express 3.0 supports ~985 MB/s per lane. While older computers with only PCI Express 2.0 can use NVMe SSDs, most NVMe SSDs use a four lane (x4) interface, limiting the drives to a theoretical peak of 2000 MB/sec, though this is in practice 10-15% lower due to overhead. This being the case, achieving the specification-listed 3400 MB/s read speeds on the aforementioned WDS100T2X0C drive would not be possible.

A related issue is noticeable in Lenovo's ThinkPad series of business-oriented notebooks. For some of the newest ThinkPads that use 8th Generation Intel processors (Kaby Lake Refresh), the system boards devote two PCIe 3.0 lanes to the SSDs, though the SSDs themselves (in this case, a Samsung PM981) are designed for an x4 interface. Samsung rates the drives for a 3200 MB/s and 2400 MB/s read/write speed, but a benchmark performed in a review of the ThinkPad P52s by NotebookCheck recorded the speeds as 1777 MB/s and 1722 MB/s, respectively. This design flaw can be found in the E480, L480, T580, and T480, while the X280, T480s, X380 Yoga, and X1 Carbon G6 correctly allocate an x4 interface to the SSD.

Additional resources


datacentervirtualizationistock-842457006vladimirtimofeev.jpg
Image: caribb, Getty Images/iStockphoto

About James Sanders

James Sanders is a Writer for TechRepublic. Since 2013, he has been a regular contributor to TechRepublic and Tech Pro Research.

Editor's Picks

Free Newsletters, In your Inbox