Ensuring Data Availability for NVMe-based Storage

End users continue to rely on RAID for server level primary storage protection. Crucial applications of RAID include cost-effective protection from drive failure for boot volumes, primary data storage and database applications.

RAID remains important for:

• OS drives (typically RAID 1)
• Primary storage in enterprise deployments (RAID 0/1/10/5/50/6/60)
• Caching for HDDs (RAID 0)

Previously, an NVMe device use has been confined to high-performance applications, due to their high cost. But, as these costs have started to subside, the ability of the Enterprise to adopt NVMe drives for traditional use models have emerged. Today NVMe SSD volumes are fast becoming the preferred storage media for capacities <4TB, according to IDC. In addition, other ecosystem barriers for NVMe primary storage have also been removed:

• Enterprise reliability (write endurance)
• Serviceability (hot plug, surprise plug)
• Standardization (Universal Bay Management)
• Technology for ubiquitous connectivity of SAS/SATA/NVMe (Tri-mode)

The removal of ecosystem barriers and the rising popularity of NVMe drives in the Enterprise server space means that RAID for NVMe SSDs will be a mandatory portfolio offering for enterprise servers by the end of the decade.

Options to Deliver RAID for NVMe

There are two system-level options to deliver NVMe RAID.

Traditional software and hardware data protection each have their advantages, including data resiliency with high IOPs, low latency, and the benefits of hardware offload are possible for PCIe-attached drives with the appropriate silicon and firmware architectures

Option 1 – Software RAID utilizes the in-box NVMe device driver on the host to access NVMe drives directly or via a switch where necessary. Utilizing the inbox driver provides the lowest possible latency path from the host to the drives, and superior i/o performances when compared to traditional hardware RAID. A detriment of software RAID is the consumption of expensive compute and memory bandwidth on the host.

• Lowest latency path
• Consumes compute and memory BW on the host CPU

Option 2 – A single path hardware RAID relieves the parity generation burden from the host. The use of single path hardware RAID funnels all data to the RAID controller hardware prior to placing them on the submission queue of the drive. Directing all data through the RAID controller, adds latency for read/write operations, even with intelligent bypass options are available to reduce unnecessary steps for various traffic patterns.

• Adding a controller to the path impacts latency for read/write operations
• Offloads compute and memory bandwidth from the host CPU

There are multiple implementations of controller-based hardware NVMe RAID.

Combining a multi-path driver with an embedded switch in the controller unlocks the best of both worlds for PCIe attached drives. The embedded switch provides a streamlined data path through the adapter unencumbered by firmware nor the raid on chip controller, all while maintaining the availability of hardware based advanced data services. The novel embodiment of the architecture is the multipath driver which intelligently manage data based on the data service requirements through either the switch or the controller with negligible overhead.

Multi-path Driver

The multi-path driver opportunistically submits commands directly to the drive submission queues via the embedded switch. When rich data services are required, such as parity or controller encryption, the multi-path driver forwards data to the value-add engines of the hardware controller. NVMe direct path operations apply to all RAID modes for single column reads to all RAID volumes or single column writes to non-parity volumes when the array is in a good state. Path selection for
commands is a nearly atomic operation and practically immeasurable in the I/O path. The subsequent command construction operates just as efficiently as a native NVMe driver in terms of instructions per cycle.

To determine which path to use, the driver performs a few lightweight checks such as:

Is the array in a good state?

Is the command a single column I/O?

Is the command a read or non-parity write?

Is the data flow plaintext?

If all states are positive, the driver issues the NVMe command directly to the NVMe drive, through theembedded PCIe switch on the controller.

The Path to Standardized Drivers

Previously, the multi-path driver talks to the controller via a proprietary protocol and presents the controller-attached NVMe devices to the host operating system as logical SCSI volumes. A more standardized approach is also possible. The full offload path to the logical volume could be represented as a logical NVMe drive attached to the controller’s PCIe switch, allowing a standard NVMe driver to enumerate and access the NVMe volume. A multi-path aware NVMe driver could then be developed to leverage the additional lower latency path directly to the drives via the controller’s PCIe switch.


Expanded use of NVMe-based devices beyond high-performance limited applications driven by the maturity of the NVMe-based storage ecosystem coupled with lowering economics, renews demand for data protection for storage. Demand for offloaded hardware RAID acceleration and rich data services have not dissipated but do require alteration from the traditional data path management to new optimized paths to maintain relevance in the NVMe market. Microsemi-based architectures deliver on
the promise of optimized data paths for protected storage utilizing the latest in embedded switch technology and innovative driver support.

Microsemi RAID adapters and controllers (ROCs) supporting NVMe architecture will sample soon.

Sign up to read more about Microsemi Data Protection over NVMe Media.

Please feel free to connect with me on LinkedIn; I am happy to answer any questions you may have.

Tags: , , , , ,

This entry was posted by Stephen Churchill on at and is filed under Storage. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply

You must be logged in to post a comment.