Storage Supply/Demand Crisis In A Zettabyte World

By Hubert Yoshida posted 10-02-2019 20:16


We are now in a zettabyte world. Last week I posted about the IDC study that predicted that the size or the global data sphere would be 175 zettabytes by 2025. The concept of a zettabyte is hard to grasp. A zettabyte is 1000 exabytes and equivalent to all the grains of sand on all the world’s beaches. In that post I described how we could manage some of that growth through better management of data copies. I also expressed doubt as to whether we would have the manufacturing capacity to meet that type of demand. Recently I found an article in Forbes by Tom Coughlin which reinforced my concern.

In this article Tom showed this growth chart from the IDC study. Notice that the global data sphere started at about 1 zettabyte in 2010 and grows to 40 zettabytes in 2019 to 175 zettabytes by 2025. It took 9 years to grow from 1 zettabyte to 40 and will take only 7 years to increase another 135 zettabytes. 

Then he showed this chart of the worldwide byte shipments by storage media types. 

If I add up the shipments per year from 2019 through 2025, I get a total of about 21.9 zettabytes, a fraction of the capacity needed to store the new demand for 135 zettabytes by 2025. On top of that most storage capacity is not allocated or allocated and not used. I could not find any recent studies, but studies back in 2013 showed that the actual utilization of capacity was about 30%. This means that of the 21.9 zettabytes of new capacity that we ship in the next 7 year period, only about 10 zettabytes would actually hold any of the 135 zettabytes of data growth from 2019 to 2015. Today most of the global data sphere resides on commercial devices but as we move to IoT we should expect to see more of that data captured on enterprise storage systems for analytics and AI/ML. 

With this tremendous shortfall in supply and demand for enterprise storage systems, you would expect storage manufacturers to be rushing to build out new manufacturing capacity. This does not seem to be the case. After a boom year in 2018, it looks like the NAND flash market is in a down swing, as an oversupply, due to high capacity of 63 layer and 96 layer 3D NAND memory devices, is starting to impact the bottom lines of memory makers. To stem any potential for significant losses or a market crash, three major manufacturers of NAND memory, Intel, Micron, and SK Hynix have announced that they will be taking measures to address the oversupply, such as reducing flash production, cutting down wafer starts, and/or slowing down ramp ups of new fabs. Another major manufacturer, Samsung, is expected to follow suit. The NAND situation has caused prices to fall by as much as 20% across multiple categories in Q1 2019, according to TrendForce. How fast can they scaleup back up and produce 10’s of zettabytes of more capacity? 

 HDD’s have a large legacy base and are still a large part of the new capacity growth. However, manufacturers are exiting this part of the storage business. HDD manufacturing is very labor intensive, and the technology has essentially run its course. The days when capacities of HDDs were doubling every 2 years are over. HDD densities will not be able to match the higher densities that will be capable with NAND and SCM technologies, making it unlikely to see significant new investments in HDD manufacturing plants. 

On September 19, Western Digital one of the few remaining manufacturers of storage devices sold off their IntelliFlash data center systems portfolio to DDN, as part of Western Digital’s strategic intention to exit Storage Systems. This may be a good move if they concentrate on manufacturing storage media rather than storage systems. After the announcement, Western Digital stocks were rewarded by investors, gaining 2% in after-hours trading. Exiting storage systems appears to be a bullish move.

Cloud storage may give us a false sense of security. We can always park data in the cloud and there is a shifting preference to low-cost original design manufacturer storage boxes, that are offered by the hyper cloud storage vendors. But ODM will have the same supply constraints, especially as more data moves to the cloud and more copies are required in the cloud for availability, distribution, and governance.

I am not a financial person nor a manufacturing expert, but if we believe IDC’s growth charts, it seems to me that we are headed into a storage supply/demand crisis and this generates a lot of questions. How will we cope with this demand? Will we need to change the way we use data? Can we extract the value than throw away the data? Can container storage help to address the problem? What do you think we will be paying for storage in 2025? What if the hyper cloud storage vendors were to buy up all the storage capacity and controlled the price of storage?

Let me know your thoughts.

1 view