All-flash continues to be the fastest growing segment of the external storage market with a 29% projected growth rate though the end of the decade (source: IDC, External Storage Systems and the Impact of Flash, October 2015). Up until to now, most all-flash arrays (AFA’s) are deployed in modest capacity sizes of 50TB or less and run just a few or even a single application. Virtual desktops and transactional databases are the primary use cases for AFAs.
So what is going to drive the hyper-growth of flash in the enterprise storage market over the next few years? First, flash is going to be even faster and less expensive than today’s technologies allow and flash deployments are becoming much more robust which will allow flash to expand far beyond today’s appliance-like deployments making all-flash data centers a reality. Second, new use cases driven by IoT are poised to generate massive amounts of data and then analyze that data in real-time as soon as it can be ingested. A flash infrastructure will be required to provide the real-time analysis that will enable data driven decisions to be made quickly and accurately. The convergence of these two trends are going to enable new use cases that have never been possible before.
Of course, flash is extremely fast as compared to traditional hard drives even those that run at 15K rpm speeds. However, flash, particularly SSDs, can suffer from unpredictable latency spikes particularly when existing data needs to be moved to make room for new data. New technologies such as 3D XPoint memory (replaces NAND) and NVMe/PCIe interfaces (replaces SCSI) promise to greatly accelerate the response times of flash devices and flash storage systems. Response times are predicted to improve by 100 times or more in the next 4 years.
But faster flash technology in and of itself won’t be enough to enable the creation of revolutionary use cases. Flash will need to be integrated into a robust infrastructure with the following characteristics:
- Massive scalability – either by scale-up or scale-out architectures so flash deployments can grow from tens of TBs to tens of PBs
- Always-on availability – because there’s simply no room for any down time in many of these future use cases
- Management automation – manual administration of day to day tasks cannot be a bottleneck if all-flash data centers are going to reach their full potential
- Rock solid security – as the value of data grows it become more attractive to hackers and, therefore, must be fully protected
- Cloud tiering – as data ages and loses value it doesn’t need to consume capacity in a flash infrastructure. Automatically tiering data by policy to a cloud infrastructure when it ages can be a very cost effective solution.
Hitachi has been a leader in large scale flash deployments because we are a leader in delivering these capabilities. Our largest flash customer in 2015 installed 16 Petabytes (PB) and we had 20 customers deploy at least 1PB of our flash last year. These customers use our Hitachi Accelerated Flash installed into our Virtual Storage Platform systems which are managed and protected by our software defined infrastructure solutions. This is a very different deployment model than found with most AFAs but one that is far more closely aligned with the demands of the all-flash data centers of the future.
Massively scalable and highly robust all-flash data centers are enabling new use cases that have never been possible before. To contribute to this trend, Hitachi made a major announcement regarding about the statement of direction for our IoT strategy last month with the formation of the Hitachi Insight Group. We’re already up and running in this space with 33 solutions in-market which generated $5.4B in revenue to Hitachi Ltd. last year. We also announced our new Lumada platform which brings proven Hitachi technologies together to synthesize actionable insights for better decision making.
One example of an area where our flash infrastructure and IoT solutions are coming together is in City Data Exchanges. A City Data Exchange aggregates data from separate repositories, both public and private that would otherwise sit in silos, and creates a shared hub of information. An example is data collected from energy consuming devices such as cars, lights, and air conditioning units can be analyzed in real-time along with the output from air quality sensors so that impactful steps can be taken to reduce pollution. Analyzing that much data so quickly can only happen on an all-flash infrastructure.
By placing so much of our development efforts in flash acceleration, software defined infrastructure management and powerful analytics, Hitachi is making the all-flash data center and innovative new use cases a reality!