Flash Storage​

 View Only

Epic Performance with Hitachi VSP 5600-2N and Hitachi Thin Image Advance (HTIA)

By Sumit Keshri posted 04-07-2023 00:53


Epic Performance with Hitachi VSP 5600-2N and Hitachi Thin Image Advance (HTIA)

Hitachi Virtual Storage Platform 5600 (VSP 5600) is a high-end, enterprise storage system designed to provide high performance, scalability, and reliability to support the most demanding workloads and applications. It offers advanced data management features, including data protection, replication, and analytics, and is suitable for use in large data centers and cloud environments. The VSP 5600 storage system is a part of the Hitachi Virtual Storage Platform (VSP) family and is known for its flexibility and ease of use.

Epic provides software that stores patient records for midsize and large healthcare organizations. In Epic software environments, patient data is stored in a single database that requires high-performance all-flash storage for files belonging to the primary database instance.

In Hitachi Vantara lab testing, the VSP 5600 achieved phenomenal Epic benchmarked performance. So, for customers looking to improve the overall performance of their Epic environments, the VSP 5600 is a proven asset to achieve significantly higher performance as shown in benchmark testing.

This article provides valuable insights into Epic GenIO and how it performs with the VSP 5600, and the improvements associated with Thin Image Advance technology that can help readers make informed decisions about implementing and managing enterprise storage systems in their organizations.

In lab testing, the VSP 5600 achieved a peak performance of 619,300 IOPS with Epic’s GenIO benchmark tool. Also, additional Epic GenIO tool testing with the new Hitachi Thin Image Advance (HTIA) feature on the VSP 5600 achieved a breakthrough performance. We are confident that the new HTIA feature addresses previous concerns about storage backup performance during SPLIT operations, a window that influences the backup SLA with customers.

During the SPLIT operation, which involves snapshot creation, HTIA testing achieved 2.46 times higher IOPS compared to standard Hitachi Thin Image (HTI) in a VSP 5600-2N setup. HTIA significantly improves the effectiveness of Epic database backup operations as well as space efficiency, allowing customers to save cost. Additionally, the performance of the Epic GenIO workload can be enhanced by creating multiple HTIA snapshots.

The VSP 5600 benchmark testing performed by Hitachi Vantara’s Center for Performance and Innovation validated that Epic customers looking to upgrade can achieve significantly improved performance with Hitachi’s latest solutions.

Introduction to Epic and Electronic health record system

Hitachi has an established relationship with Epic as one of the elite group of vendors who have successfully implemented effective storage solutions for customers.

Epic primarily develops, manufactures, licenses, supports, and sells a proprietary electronic medical record software application commonly known as 'Epic' or an Epic EMR. An electronic health record (EHR) is a digital version of a patient’s health records. EHRs are real-time, patient-centered records that make information available instantly and securely to authorized users. EHR is one area where a large amount of data must be stored efficiently to allow rapid access to patient data when and where they need it. Because of the sensitivity of this data, records must be stored in a secure and redundant manner.

Epic developed and uses the Generate IO tool (GenIO) to ensure that a storage solution is product ready for customers to implement and utilize.


GenIO workload profile and validation parameters

The GenIO tool simulates workload profiles that are commonly seen in a real-life Epic production environment. From a storage standpoint, the tool simulates many reader processes with random read operations using a block size of 8 KB. These readers mimic end-user processes, making the random read latency critical to end-user performance.

o    The read-to-write ratio is usually three reads to one write (75% Read). The simulator does not write directly to the underlying database files. Instead, it writes to the memory, with eight write daemons responsible for flushing out dirty data in memory to the files on the drives. The eight write daemons are active every 80 seconds, flushing the data out to the drives.

o    The write burst must complete within 45 seconds (the maximum value accepted for ‘Longest Write Cycle’) to avoid dirty block accumulation. The following events occur during the write burst:

       At the beginning of the write burst, one of the eight write daemons writes all the dirty data to a single file called WIJ. The writes to the WIJ file are sequential, while the block size can be 256 KB.

       After all sequential writes to the WIJ file are complete, the eight write daemons then flush out the same dirty data to different database files on drives. The writes to the database files are random and 8KB in size (similar to the reads).

o    The dirty data is written twice every 80 seconds, once to the WIJ file and a second time to the corresponding database files.

Criteria for a successful and valid test result using GenIO tool

       Average Read Response time: Within 2 milliseconds.

       Longest Write Cycle: Within 45 seconds

       Distribution of Random read latency: 99% < 60 ms, 99.9% < 200 ms and 99.99% < 600 ms



Performance validation on Hitachi VSP 5600 storage system

o    When deploying an Epic environment leveraging a VSP 5600-2N storage system (with FCP protocol, encryption enabled, 48 NVMe drives, RAID-6 6D+2P, and 32 front-end ports) using four hosts, the system was pushed to its peak performance capabilities for an Epic deployment. The optimally configured and tuned system achieved 619,300 IOPS with an average read response time of 1.4ms. In a non-ADR (Adaptive Data Reduction) setup, the longest write cycle was 44.57 seconds.



o    In an ADR (Deduplication + ACLF Compression) setup, we achieved 381,300 IOPS with an average read response time of 0.85ms and the longest write cycle of 44.19 seconds using a similar storage system configuration.

In both non-ADR and ADR configurations, using NVMe-oF resulted in similar throughput as FC.



Comparison of VSP 5600-2N Non-ADR and ADR Epic GenIO Performance results with VSP 5200-2N, E990, and E770 storage systems


VSP 5600 storage system in a non-ADR setup achieved:

       2.35x better IOPS compared to VSP 5200 storage system

       3.24x better IOPS compared to VSP E990 storage system

       3.75x better IOPS compared to VSP E790 storage system

VSP 5600 storage system in an ADR (Deduplication + ACLF Compression) setup with 49% capacity saving achieved:

       2.4x better IOPS compared to VSP 5200 storage system (ADR Deduplication + ACLF Compression)

       2.8x better IOPS compared to VSP E990 storage system (DKC Compression + Deduplication)

       2.9x better IOPS compared to VSP E790 storage system (DKC Compression + Deduplication)

Introduction to Hitachi Thin Image Advance (HTIA) snapshot

With the latest release of SVOS, Hitachi introduced a new feature called Hitachi Thin Image Advance (HTIA). This feature uses redirect on write snapshot technology to help customers significantly reduce the latency caused during split operations in an ADR setup. After performing the PAIR SPLIT operation with Thin Image in an ADR setup, we discovered that the storage system was unable to achieve better IOPS while maintaining a low response time, which was one of the performance issues with Thin Image setup that has now been resolved with the availability of HTIA.

HTIA is used for logical backup, repurposing data, data warehouse, and recovery against logical corruptions or ransomwares.  It is currently supported in an ADR setup with a compression or deduplication and compression setup.

As we all know, we create a Data Reduction (DRD) volume in Thin Image; however, in Thin Image Advance, we create a new type of volume called Data Reduction Shared (DRS) volume. Both P-VOLs and S-VOLs must be DRS volumes in the same DP-pool.

       Performance improvement:

o    8kB granularity Redirect-on-Write Snapshot offers a benefit for random write performance during split from Copy-After-Write or Copy-on-Write (CaW/CoW) technology (Thin Image snapshot). 

o    Snapshot S-VOL performance improved compared to CoW based Thin Image snapshot S-VOL access.

o    Metadata aware copy technology allows faster restoration (reverse resync) than the current CaW/CoW based Thin Image snapshot. 

o    HTIA provides space efficiency for snapshot data, allowing customers to save cost.

o    Workload performance can be improved by creating multiple HTIA snapshots.

       VSP 5000 series is the world’s first true RoW snapshot technology in the enterprise segment.

o     IBM (DS8000 FlashCopy) and Dell (PowerMax SnapVX) are based on CoW technology.

o    PowerMax SnapVX is performed in Async Copy-on-fast-write (ACOFW) mode if write IO is less than 128kiB or misaligned 128kiB SRP chunk.



Comparison of HTIA versus HTI snapshot in an ADR (Deduplication and ACLF Compression) setup



As shown by the data point in the graph, we achieved 2.46x higher IOPS in HTIA when compared to HTI during Split operation, which involves the creation of a snapshot.

In HTIA PAIR Suspended (Split) state, data shared by P-VOLs and S-VOLs does not need to be invalidated; instead, P-VOLs data pointers are redirected to the S-VOLs when the existing data is deleted or overwritten.  So, because the overhead is less in PSUS (Split) state than in the simplex or PAIR state, the throughput is higher in HTIA when compared to the HTI setup.


The following image shows how the HTI and HTIA snapshot technology works when write operations are invoked in P-VOLs:

HTIA snapshots are particularly useful in storage operations because they enable instant creation or restoration of backups and replicas of an Epic database without affecting the original database. This indicates that backup and replica operations can be performed without locking or blocking access to the primary database. HTI offers similar benefits; however, the restoration window is longer than HTIA for the same database size.

Furthermore, when compared to HTI, HITA snapshots can improve the storage operations performance by reducing the amount of data that must be transferred and stored. Instead of transferring or storing the entire Epic database, HTIA saves the pointers to changes made since the last snapshot was captured.

Overall, HTIA snapshot is an important feature in storage systems that can significantly improve the effectiveness of Epic database backup operations as well as space efficiency, allowing customers to save cost.

Customers who are looking to improve their Epic deployment with HTIA must consider various factors such as data security, redundancy, performance, space efficiency, and ransomware protection.

·         Data Security: Epic deployment contains sensitive data such as patient records, financial information, and other confidential data. Therefore, securing this data is critical to avoid data breaches and other security incidents. To improve data security, customers can implement various security measures such as data encryption, access controls, and keeping the storage microcode up to date. Hitachi Data Protection Suite (HDPS) is a good bet!

·         Redundancy: Epic deployment is a critical system that must have high availability and fault tolerance. Customers can improve database access redundancy by implementing fault tolerance architecture with multiple servers and Hitachi storage failover mechanisms. This ensures that if any front-end servers or storage components fail, the system continues to function uninterrupted.

·         Performance: HTIA helps us achieve better IOPS while maintaining a low response time after performing SPLIT operation, which was one of the limitations with HTI. To maintain and improve performance in a Hitachi storage system, customers can monitor performance, optimize the storage system and server along with database queries, and then apply indexing and caching techniques. They can also improve the infrastructure by upgrading to Hitachi VSP 5200 or 5600 storage systems and optimizing the operating system.

·         Space efficiency: Hitachi storage systems offer 49% capacity savings with ADR (Deduplication + ACLF Compression) configuration for the data in primary volumes. Additionally, the HTIA feature provides space efficiency for snapshot data, allowing customers to save cost. The following figure shows how Hitachi stored inline storage efficiency with default Epic GenIO parameter:



o    Raw capacity is the total capacity of drives before creating parity groups.

o    Usable pool capacity is the total pool capacity we get after creating parity groups, pool volumes, and pool.

o    Effective capacity is the maximum capacity that can be allocated to the host or client in an ADR (Deduplication + ACLF Compression) setup.

Note: Capacity savings were measured using the data filled from the Epic GenIO tool. In production, Epic incorporates database consistency routines to perform system health checks, and enabling ADR can result in a considerable increase in system load. However, Epic customers have had to be selective in their use of ADR, and the savings have limitations.

·         Ransomware Protection: Ransomware attacks are a growing concern for IT departments. Customers can protect their Epic deployment from ransomware attacks by implementing a comprehensive HTIA backup strategy, monitoring for suspicious activities, and performing regular microcode upgrades.


For more information, see: VSP 5200-2N Epic Test Results (Non-ADR and ADR Performance)