Flash Storage​

 View Only

FC-NVMe and SCSI-FCP Reality versus Fiction

By Stephen Rice posted 12-23-2022 12:00


There seems to be some confusion about the difference between SCSI-FCP and the enhancements of FC-NVMe.

Let's start at the beginning for a quick recap. Both FC-NVMe and SCSI-FCP use the Fibre Channel Protocol (FCP). Fibre Channel standards are still enforced, the frame layout, and mapping of each into FCP is still the same, and the payload of a Fibre Channel frame is still a little over 2k (2112 bytes) regardless of which protocol uses FCP.

So where is the difference?

The difference between FC-NVMe and SCSI-FCP lies within the Fibre channel frame in the contents of the “Optional headers, Commands, and Data Payload”. However, the architecture and sizes of SCSI-FCP and FC-NVMe Commands, Transfer Ready Signals, and Responses remain very similar. In addition, the round-trip journeys and sequences required to complete both reads and writes are the same.

FC cut-through switches do not verify any section of the frame to reduce latency. However, FC store and forward switches ensure that the frame is complete, and the Cyclic Redundancy Check (CRC) is correct. As a result, neither switch methodology opens the FC frames to determine what protocol is being sent because that is included in the optional headers or data payload section of the frame.  Furthermore, most FC switches are not concerned with the internal protocol.

Both FC-NVMe and SCSI-FCP use a frame interleaving architecture as shown in the following block diagram:

The Host System (purple) requests that both disks in the storage subsystem, the blue and the red SCSI LUN or NVMe Namespace, read some information. Fibre frames from both disks are interleaved, with red and blue disk Fibre frames flowing through the switch back to the host.

To continue the discussion of frame interleaving, the following block diagram depicts a single purple host requesting a read from both the blue and green storage subsystems.


The host system (purple) is again receiving read data from the blue, red, light green, and dark light green disks (SCSI LUN or NVMe Namespace), all of which are interleaved with each other as they pass through the switch and head to the host.

In the current generation of the host systems, HBA can use either FC-NVMe and/or SCSI-FCP. Therefore, there is no hardware advantage in the HBA, or the HBA driver software.  

The speed of the Fibre channel remains constant between FC-NVMe and SCSI-FCP. Therefore, there is no reduction to latency because of distance. In addition, the transmission of light through the Fibre channel remains unchanged.

To quickly net it out, Fibre channel remains constant, and supports multiple protocols.

The advantage of FC-NVMe is in queuing. FC-NVMe is designed only to reduce I/O latency caused by queuing delays. FC-NVMe can provide 64K queues per port, each capable of holding 64K commands. At connect time, a host can request multiple FC-NVMe queues, such as one pair of submission and completion queues per CPU. The architecture allows for a smaller number of larger volumes with more I/O parallelism.

We set up a quick experiment to illustrate the advantage of NVMe.  The experiment was carried out with two hosts, each with two ports and four very large LUNs. In a SCSI-FCP environment, smaller LUNs with less queuing and smaller LUNs with fewer IO to each LUN would be used. NVMe appears to have no issue handling the large queue depth that the experiment was designed to create.

The following diagram shows that there is less need for queuing tuning in the NVMe environment.

FC-NVMe is a new and emerging technology with the ability to self-tune the IO environment.  While we have been manually tuning the queue depth on the SCSI environment for 40 years and SCSI-FCP for the last 20 years, manual tuning exercise should be a skill of the past with FC-NVMe. This allows for fewer and larger disks (SCSI LUN or NVMe Namespace) to be connected to the host subsystem.

Because FC-NVMe is a new and emerging technology, now might be the right time to start considering new technologies to purchase an “FC-NVMe compatible” system. Most systems can still run in SCSI-FCP mode until you are ready to switch to the FC-NVMe protocol.