Introduction to Virtual Storage Platform E990 Architecture

By Charles Lofton posted 09-24-2020 22:00



The Virtual Storage Platform (VSP) E990 is Hitachi’s newest midsized enterprise platform, designed to utilize NVMe to deliver industry-leading performance and availability. The VSP E990 features a single, flash-optimized Storage Virtualization Operating System (SVOS) image operating on 56 processor cores, sharing a global cache of 1 TiB. Thanks primarily to SVOS optimizations, the VSP E990 offers higher performance with fewer hardware resources than all its competitors. VSP E990 cache architecture has been streamlined, permitting read response times as low as 64 microseconds. Improvements in reliability and serviceability allow the VSP E990 to claim an industry-leading 99.9999% availability (on average across the installed base, 0.3 seconds per year of downtime expected). In this blog post, we’ll take a brief look at the highlights of the VSP E990 architecture.

Engineered for Performance

The VSP E990 features a new controller board powered by two 14-core CPUs operating at 2.3 GHz. The two CPUs function as a single 28-core multi-processing unit (MPU) per controller. The upgraded controller board adds 40% more CPU power compared to the VSP F900. The extra processing power enables the VSP E990 to leverage the NVMe protocol, which has a low-latency command set and multiple queues per device, for unprecedented performance in a midsized package. Because NVMe doesn’t allow for cascading connections, the VSP E990 supports a maximum of 4 NVMe drive boxes, connected to the controllers via PCI Express Gen3. This simple, streamlined configuration allows for low-latency, point-to-point connections between the E990 controllers and the NVMe SSDs.

Figure 1. The VSP E990 series offers flexible configuration options with industry-leading performance and 99.9999% availability

The VSP E990 Controllers

Figure 1 presents a block diagram of the VSP E990 dual-controller system. The controllers have a central role in connecting all of the other VSP E990 system components and allowing them to function as one unit. Each channel board (CHB) or NVMe disk board (DKBN) is connected to a controller via 8 x PCIe Gen 3 lanes, and thus has 16 GB/s of available bandwidth (8 GB/s send, and 8 GB/s receive). Pictured above is the configuration with two pairs of NVMe disk adapters, which supports 1-2 NVMe drive boxes (DBNs) and up to 48 NVMe SSDs. Each controller in the pictured configuration therefore has up to 96 GB/s of front-end theoretical bandwidth (if six CHBs per controller were configured), and 32 GB/s of back end bandwidth (two DKBNs per controller). An alternative configuration (not pictured) doubles the capability of the NVMe back end, having four DKBNs per controller and supporting 3-4 drive boxes and up to 96 NVMe SSDs. The latter configuration would have 64 GB/s of back end bandwidth per controller, and up to 64 GB/s of front-end bandwidth per controller (if 4 CHBs per controller were installed).  


Like previous Hitachi enterprise products, all VSP E990 processors run a single SVOS image and share a global cache. Cache is distributed across individual controllers for fast, efficient, and balanced memory access. Although VSP E990 hardware and microcode will permit a variety of cache configurations, the only configuration available to purchase has maximum cache (1 TiB). Therefore, all eight DIMM slots per controller will be populated with 64 GB DDR4 2133 MHz DIMMs for a total of 132 GB/s of theoretical memory bandwidth per controller.

Hardware Components

While the VSP E990 has a new and faster controller board, other basic hardware components are shared with the VSP 5000 or VSP F900. The fibre channel and iSCSI front end adapters (CHBs) are shared with VSP F900. Up to six four-port 8/16/32 Gb FC or six two-port 10 Gb iSCSI CHBs per controller may be installed. (Protocol types must be installed symmetrically between controller 1 and controller 2). Disk adapters for the all-NVME back end are shared with the VSP 5000, as are the NVMe SSDs, which are available in four different capacities (1.9 TB, 3.8 TB, 7.6 TB, and 15 TB). The NVMe drive box is also shared with VSP 5000. However, unlike the VSP 5000 which has strict rules about Parity Group configuration, the VSP E990 drive box can be ordered in quantities as small as a single tray, and parity groups can be created from drives in any drive slot locations. 

Front End Configuration

VSP E990 FC ports operate in universal (also called bi-directional) mode. A bi-directional port can simultaneously function as a target (for host I/O or replication) and initiator (for external storage or replication), with each function having a queue depth of 1,024.  The highest-performing VSP E990 front end configuration would use “100% straight” access, in which LUNs are always accessed on a CHB port connected to the controller that owns the LUN. Configuring for front end straight I/O reduces ease of use for initial configuration and when load balancing by changing the owning controller, so it is not recommended unless the customer requires the highest possible levels of performance. Addressing a LUN on the non-owning controller (known as “front end cross” I/O) incurs a small additional overhead on each command. However, our testing shows that front-end cross I/O does not have a significant performance impact under normal operating conditions (up to about 70% controller busy). Configuring for round-robin multi-path I/O is recommended for most cases because it results in a good balance between performance (on average 50% front end straight I/O), and ease of use for configuration and load balancing (not having to align a primary path to the owning controller for each LUN).

Figure 2. VSP E990 Universal Port Functionality

A front end I/O expansion module (a common component with F900) is also available for VSP E990. As shown in Figure 3, two CHB slots per controller may be used to connect to as many as four CHBs per controller in the expansion module. With the expansion module in place, a diskless VSP E990 (serving as a powerful virtualization engine) could present up to 80 FC ports, or 40 iSCSI ports per system. But note that the eight CHB slots in the expansion module must share the PCIe bandwidth of the four slots to which the expansion module is connected, which might limit throughput for large-block workloads.

Figure 3. The I/O Expansion Module Permits Installation of Up to Ten CHBs Per Controller

Back End Configuration

The VSP E990 has an all-NVMe back end, which makes configuration relatively simple and straightforward. Either two or four disk adapters per controller may be installed. As presented in Figure 4, either channel boards or disk adapters may be installed into slots 1-E/F and 2-E/F in each controller. A configuration with two disk boards per controller can support one or two NVMe drive trays, and up to 48 NVMe SSDs. With four disk adapters per controller, three or four drive trays can be connected, accommodating up to 96 SSDs (see Figure 5).  Each NVMe disk adapter has two ports, which are connected to two different NVMe drive boxes via 4-lane PCIe Gen3 copper cables, as shown in Figure 5. Each cable connection has 8 GB/s of PCIe bandwidth (4 GB/s send, and 4 GB/s receive). Each NVMe drive tray with its standard 4 x four-lane PCIe connections therefore has 16 GB/s send and 16 GB/s receive of PCIe bandwidth. Within each drive box (Figure 6) are two PCIe switches, each of which is connected to two NVMe disk adapters via 4-lane PCIe cables. As shown in Figure 6, each NVMe SSD is connected to both PCIe switches via the drive tray’s backplane. In summary, each NVMe SSD can be accessed via a point to point PCIe connection by two different NVMe disk adapters from each controller, for a total of four redundant back end paths per drive.

Figure 4. Multi-Purpose Slots Permit Installation of Two or Four DKBN Pairs

Figure 5. Connection Diagram of the Maximum Back End Configuration

Figure 6. DBN Block Diagram

Due to its positioning as a mid-sized enterprise array, flexible Parity Group configuration is available on VSP E990. Table 1 shows the supported Parity Group configurations, which can be used on any combination of 1-4 drive trays.

Table 1. Supported Parity Group Configurations

Finally, encrypting disk boards (eDKBNs) are optionally available for the VSP E990. The eDKBNs offload the work of encryption to Field Programmable Gate Arrays (FPGAs) as shown in Figure 7. The FPGAs allow FIPS 140 level 2 encryption to be done with little or no performance impact. Encrypting DKBNs are also recommended for customers requiring the maximum non-ADR[1] sequential read throughput of 40 GB/s (which is only available in configurations having at least three drive trays). The eDKBNs optimize PCIe block transfers, thus requiring fewer direct memory addressing (DMA) operations and improving non-ADR sequential read throughput. This additional throughput potential is possible regardless of whether encryption is licensed or enabled, making it available in export restricted countries. With the non-encrypting DKBN, the non-ADR sequential read throughput is 29 GB/s. 

Figure 7. eDKBN Block Diagram

Performance and Resiliency Enhancements

Significant enhancements in the VSP E990 include:

  • An 80% reduction in drive rebuild time compared to earlier midsized enterprise platforms.

  • Smaller access size for adaptive data reduction metadata reduces overhead.

  • Support for NVMe allows extremely low latency with up to 5X higher cache miss IOPS per drive.

  • Upgraded controllers have 40% more processing power than VSP F900.

We’ve briefly reviewed the highlights of VSP E990 architecture, including improvements in performance, scalability and resiliency. For additional information, please visit the VSP E990 page.

[1] “Non-ADR” refers to a configuration in which the adaptive data reduction feature (ADR) is not used.

1 comment



11-04-2020 10:12

Thanks for the information keep sharing such informative post keep suggesting such post myaccountaccess .