Skip navigation
1 2 3 Previous Next

Hu's Place

281 posts

GAD Brains.png

The days when data recovery solutions were evaluated on how they could minimize the two R’s are over. The two R’s stand for Recovery Point Objective, RPO, how much new or changed data is lost because it hasn’t been backup yet, and Recovery Time Objective, RTO, how long it takes to resume operations. The reason I say this is because we can achieve zero RPO and zero RTO with today’s Global-Active Device on our family of Hybrid G Series and All Flash F Series Virtual Storage Platform, VSP, storage arrays.

 

The Global-Active Device (GAD) is another virtualization capability of the VSP storage platform that creates a virtual storage machine. A virtual storage machine virtualizes two separate VSP storage arrays and makes them appear as one storage array to a host server or a cluster of host servers. GAD is configured so that the primary and secondary storage systems use the actual information of the primary storage system, and the global-active device primary and secondary volumes are assigned the same virtual LDEV number in the virtual storage machine. This enables the host to see the pair volumes as a single volume on a single storage system, and both volumes receive the same data from the host. When a write is done to one of the GAD pair volumes, the data is replicated to the other pair and acknowledged before a write complete is returned to the initiator of the write. That keeps the volumes in synch and ensures zero RPO and zero RTO in the event of a storage system or site failure.

 

The virtual storage machine can span across physical storage systems that are separated by up to metro distances (500 kilometers). GAD provides non-disruptive, high availability (HA), disaster recovery (DR), and rapid data center migration services. In addition, it enables painless virtual machine storage motion where the location of the storage underlying virtual machines is moved between storage environments for load balancing or maintenance. GAD supports active-active processing of shared data, meaning that all interfaces to the storage system are always active, and the system synchronizes writes across the entire storage system. GAD can be configured in a single or clustered server configuration.

 

GAD Config.png

 

Some vendors might use their synchronous replication product to create a synchronized copy pair. This creates an Active/Passive pair on separate storage arrays. Processing occurs on the primary side of the pair and the passive side is only used for fail-over. This does not provide active-active processing of shared data. In order to achieve active-active processing, GAD provides three key features that distinguishes it from synchronous replication products.

 

  1. The first is a streamlined locking mechanism between the storage systems to synchronize the writes.
  2. The second is preservation of SCSI reserves, a control mechanism for avoiding conflicts or congestion with different host initiators.
  3. The third is a quorum disk to resolve conflicts between the active-active pair of storage systems.

 

The quorum disk is an external storage system that acts as a heartbeat for the GAD pair, with both storage systems accessing the quorum disk to check on each other. A communication failure between systems results in a series of checks with the quorum disk to identify the problem for the system to be able to receive host updates. The external quorum disk is accessed like a virtualized external disk which means that it can be a third party disk. It can also be on a virtual or physical server that has software to present itself as a disk. The quorum disk can also be in the cloud through the use of iSCSI. However, if GAD is used simply for non-disruptive migration, no quorum disk is needed.

 

GAD is a copy technology and can be one of multiple overlapping data copy technologies that are used in a production system. Backup and point in time copies are still required to protect against data corruption caused by errors or malicious attacks. While GAD provides protection in a metro area, an asynchronous, out of region replication with Hitachi Universal Replicator (HUR) might also be required in addition to GAD. Copy technologies include synchronous and asynchronous copies, clones, snap shots, and thin images or virtual copies. Copies are not only used for operational and disaster recovery, but also for archive, big data, analytics, audits, e-discovery, development and test, and other data repurposing requirements. IDC estimates that 60% 0f corporate data consists of copies of original data, and that on average, organizations have 13 copies of each data object. Managing the life cycle of copies is a major challenge. Unmanaged or orphan copies of data can be a liability. Hitachi Vantara’s Hitachi Data Instance Director provides a copy data management platform that simplifies creating and managing policy-based workflows that support business functions with controlled copies of data. Integration with Hitachi Data Instance Director is a key advantage of Hitachi Vantara’s GAD.

 

HDID with GAD.png

Hitachi’s Global Active Device is the leader in active-active, continuous availability. Other storage vendors may claim similar replication capabilities. Since they all rely on synchronous replication over metro distances there is not much difference in performance. However, there is a great deal of difference in simplicity, scalability, and ROI.

 

  1. Simplicity. Hitachi’s implementation is simple in that it is an extension of the virtualization capability of our Storage Virtualization Operating System RF (SVOS RF). There are no additional appliances required to virtualize the pair of devices or additional software required in the host servers or virtual machines. No BIN files are required to set the configuration. Our virtualization also provides the most choices for selection of a quorum device. Integration with Hitachi Data Instance Director makes it easy to manage the interactions with other data copy technologies and application requirements.

 

  1. Scalability. The way we have stream-lined the synchronous replication of the GAD pairs and preserved the SCSI reserves provides greater scalability than competitive implementations. Competitive solutions that require the use of appliances scale by adding more appliances and this also adds more complexity. Hitachi GAD provides true active-active processing of shared data, across the storage arrays. This scales beyond other implementations where only one storage array is active while the other is used for standby. The controllers in each VSP storage array are also active-active which gives us greater scalability than other vendor’s controllers that are Active/Passive or ALUA (Asymmetric Logical Unit Access). (see my previous blog on the differences in storage array controllers)

 

  1. ROI. The return on your investment is higher with GAD due to our overall virtualization capability to virtualize external storage and create virtual storage machines. You can virtualize your existing storage behind a VSP array and create a virtual storage machine with another VSP over metro distances. Neither VSP in this scenario requires any capacity from a Hitachi storage system, all the capacity can be from virtualized 3rdparty storage systems. All VSP systems, from midrange to high-end enterprise support GAD, so you don’t need to have different solutions and different management tools for midrange, entry enterprise or high-end enterprise. No additional appliances or host software is required to support GAD. GAD also integrates well with other copy requirements such as HUR and thin images since it is a feature in SVSO RF. The use of Hitachi Instance director with GAD also reduces the cost of copy management and ensures that resources are not wasted on copies that are no longer needed.

 

Learn More

 

For more information on Hitachi Vantara’s GAD implementation

High end enterprise storage systems are designed to scale to large capacities, with a large number of host connections while maintaining high performance and availability. That takes an architecture where a host can access any storage port and have direct access to a number of storage controllers which can service I/O requests and process their data services requirements. This takes a great deal of sophisticated technology and only a few vendors can provide such a high end storage system. Hitachi’s, high end, Hybrid Virtual Storage Platform (VSP) G1500 and All Flash VSP F1500 arrays provide this capability which is made possible through a switch architecture which dynamically connects front end ports with controllers and a global cache with backend device controllers.

VSP Family.png

For the midrange user where cost is a key factor and massive scalability is not required, the architecture has to be changed to trade off scalability for reduced cost. However, that trade off often means that some advanced enterprise functions like distance replication and zero data loss recovery would be compromised. With recent advances in Intel multicore technology and PCIe extensions, it is now possible to scale down these systems to reduce costs while still providing these advanced enterprise functions. The recent models of the Hitachi Vantara VSP Hybrid G and All Flash F series storage arrays, now provide, cost efficient, scaled down versions of the G1500 and F1500 Enterprise storage arrays, without any loss in enterprise functionality. This was done by consolidating the VSP controller, cache memory, and interconnects onto a cost optimized, two controller architecture which utilizes Intel’s latest multicore processors with high speed PCIe interconnects. These two controllers are configured as Active/Active controllers, which means that any I/O can access any device through either controller.

 

Midrange storage arrays are configured with dual controllers for availability. However, there is a great deal of difference in how the two controllers are configured. Most are configured as Active/Passive or Asymmetric Logic Unit Access Asymmetric (ALUA). Very few are Active/Active. The configuration of the storage controllers is a key differentiator when it comes to the performance and functionality of the storage system. Here are the differences between the different types of controller configurations.

Active Passive.png

 

ALUA.png

 

Active Active.png

 

The VSP’s Active/Active configuration is made possible through the concatenation of the cache that is attached to each controller. Both controllers work with the same cache image of the storage LUN. This configuration is also known as Active/Active Symmetric since both controllers can process the I/O request versus the prior ALUA asymmetric configuration above. This requires additional routing information that is provided by the Storage Virtualization Operating System that is available in all models of the VSP series. This Active/Active capability provides many benefits. We don’t need to worry about LUN ownership as in the case of asymmetric controllers. This provides the ability to use VMotion for live migration of running virtual machines between servers that don’t have identical primary failover controller assignments, with zero downtime, continuous service availability, and complete transaction integrity. This also provides the ability to load balance across a SAN without the worry of creating a performance issue on the storage. Since the VSP can virtualize external storage, this also makes it possible to process a cache image of a LUN from an externally attached storage system. This ability to virtualize external storage enables the extension of the useful life of legacy storage systems and the non-disruptive migration of data to and from an external systems as I posted in a previous blog.

 

The latest versions of our midrange offerings, the Hybrid VSP G350/G370 and All Flash VSP F350/F370 come in a cost effective 2U of rack space with the enterprise features of our high end and over 1 million IOPS for the VSP F370. All priced and packaged for the midrange.

 

On the higher end, which many refer to as the entry enterprise, we offer the Hybrid VSP G700/G900 and All Flash VSP F700/F900 that can scale to 2.4 million IOPS and comes in 4U of rack space. The differences in the model numbers is based on the number of Intel processors and cores. Both Hybrid and All Flash entry enterprise models have the same dual controller Active/Active design as the midrange models but with many more Intel cores, memory, front end ports, back end ports, and higher internal bandwidths. Here are the specifications for the Entry Enterprise Models.

 

VSP Data Sheet.png

The Hybrid VSP G1500 and All Flash VSP F1500 are included to show the full scale out capability of their switch architecture. The VSP G/F1500 is a multi-controller architecture in which the processors, memory modules, front end ports and back end ports are all connected through an internal non-blocking, cross bar, switch, which enables a LUN to be accessed directly through any port without the overhead of passing control between controllers.

Cross bar switch.png

 

There are other architectures which support more than two controllers, but those types of architectures involve a great deal of overhead and complexity. Without an internal switch architecture, no matter how many controllers you hook together, I/O requests for a LUN still have to be processed through the controller that owns the LUN. So If your I/O request comes in from another controller port, the I/O has to be passed to the controller that owns the LUN, creating more overhead and complexity with each additional controller that is added. Having two controllers adds redundancy but adding more controllers to a two controller architecture, can create less availability since the cumulative failure rate increase with each added controller. Having two controllers fail independently is rare, and a two controller failure is usually due to a base failure which would affect all the controllers no matter how many you have.

Four controller.png

 

 

In the case of the VSP G/F1500 the switch architecture would allow the controllers, cache, and LUNs to be assigned dynamically and fail independently.

 

The VSP family of storage arrays, provides a choice of cost optimized configurations from midrange to high end scalable enterprise systems, all running the same software and providing the same functionality to help our customers preserve their investments in policies and procedures and leverage their ecosystem-related investments. Although there is a difference in architectures we are able to simulate the architectural differences in software so that all the models have the same functionality even when it is scaled down to the midrange price and packaging. Our dual controller architectures are fully Active/Active which differentiates us from many other midrange and entry enterprise systems.

In my last blog post I explained how Hitachi Vantara’s All Flash F series and Hybrid G series Virtual Storage Platform (VSP) Systems can democratize storage services across midrange, high end, and mainframe storage configurations. Midrange customers can enjoy all the high end functions that are normally only available on high end systems since Hitachi’s VSP platform is powered by one Storage Virtualization Operating System RF (SVOS RF).

VSP Family.png

In that post I shared the CRN interview with Dell’s Vice Chairman Jeff Clarke, where he announced their plan to consolidate four disparate midrange products and focus on one. That plan immediately drew questions from other vendors as to how they would manage the migration when they transition from the old to the new. This plan was also limited to midrange products and did not extend to high end where features like active/active metro clustering (VPLEX) and SRDF replication are only available on their VMAX and PowerMax systems. SRDF replication is used to provide non disruptive migration for VMAX systems, but is not available on their midrange products.

In other words, their storage plans do not go far enough and do not address the key question of migration for their midrange storage. Hitachi Vantara can provide a solution for EMC Dell users by virtualizing their current EMC Dell storage arrays and democratize the use of Hitachi VSP high end enterprise storage services, making them available to all EMC DELL users, whether they are midrange, high end or mainframe. This capability is not limited to EMC Dell but is applicable to any vendor’s fibre channel or iSCSI attached storage arrays. As the name implies, the signature feature of our VSP is virtualization. Our approach to storage virtualization is unique in the industry since we do the virtualization in our Virtual Storage Platform (VSP) controllers with a Storage Virtualization Operating System RF (SVOS RF). Hitachi storage virtualization can greatly simplify storage management, particularly when used to consolidate and virtualize arrays from multiple storage vendors or multiple storage array architectures from the same vendor. We announced storage virtualization in 2004 with our Universal Storage Platform (USP). Hitachi was one of the first to announce storage virtualization and has carried that forward in our latest VSP Hybrid (G Series) and All Flash (F Series) storage systems.

VSP Virtualization.png

While other vendors approached storage virtualization through the use of appliances sitting on the Storage Area Network (SAN), Hitachi’s unique storage architecture enables the virtualization to be done in the storage controller. The advantage of this approach ensures that Hitachi storage virtualization would have the same reliability, availability, scalability, and performance of an enterprise class storage system, rather than the limited capabilities of an appliance sitting on the SAN. Externally attached, fibre channel or iSCSI, heterogeneous storage is presented as a common pool of storage to the VSP controller and benefits from all the Hitachi enterprise features of the VSP such as Active/Active Metro Clustering, three data center disaster recovery, predictive analytics, and automated management of common workflows. Other vendors’ midrange and enterprise storage, which lack these features, are immediately upgraded with these services when they attached to the VSP. Our approach to storage virtualization is a perfect complement for server virtualization and we were the first storage virtualization vendor to certify with VMware.

 

Virtualized third party storage can also benefit from VSP’s Storage Plug-in for Containers. Containers are a new form of lightweight and portable operating system virtualization without the overhead of an operating system image. All the necessary executables are inside a container; binary code, libraries, and configuration files to run microservices and larger applications. Containers were designed to be stateless which means that its disk files go away when the container goes away. In order to persist the data and make it easier to manage storage, Docker introduced a set of commands for volume management. This enables storage vendors to create plugins to expose their storage functionality into the Docker ecosystem. Hitachi Storage Plug-in for Containers lets you create containers and run stateful applications by using Hitachi VSP series volumes, including externally virtualized volumes, as dynamically provisioned persistent volumes. Hitachi Storage Plug-in for Containers utilizes built-in high- availability to enable a Docker swarm manager or a Kubernetes master node to orchestrate storage tasks between hosts in a cluster. Storage Plug-in for Containers can also be used in non-clustered environments.

 

Since external storage uses the licensed services of the host VSP, there is no need to have separate licenses for the external storage. This not only reduces license fees, but also reduces the operational cost of managing features and licenses on the attached storage systems. Virtualization can further reduce costs by extending the life of legacy systems that have already been capitalized or reduce costs by non-disruptively migrating systems when a refresh is required.

 

According to CIO.com legacy systems are the Achilles heel for digital transformation. “Unfortunately, many businesses today are hard-coded in technology investments of the past: legacy systems – containing the data that they need to make decisions today and drive insights. Now more than ever, reliance on legacy systems is one of the biggest hurdles in the digital transformation journey.”

 

When it comes to upgrading legacy infrastructure, the biggest challenge is the upgrade of storage systems. Unlike compute and network infrastructure, storage is stateful, because it has data that must be moved from the old to the new storage system. This movement of data takes time and as the amount of data continues to increase, the data migration extends into days, weeks, and even months. The least disruptive and easiest way to move data is through storage virtualization. Virtualization of legacy storage ensures that applications can continue to access the data while it is being moved in the background. The VSP's bidirectional FC and sSCSI ports allow simultaneous host and external storage connectivity without the need for dedicated "eports" and host ports. Logical device migrations are able to proceed while the original logical device remains online to the hosts and the VSP will seamlessly switch over the mapping from the source logical device to the target logical device when completed. No changes to host mapping are required.

 

Appliance based virtualization systems, can move data in the back ground, but performance, reliability, and scalability are limited by the capabilities and resources of the appliance compared to a fully functional enterprise virtualization engine with its large cache and large number of bi-directional ports. Utilizing an enterprise storage controller also leverages the built-in security features of the controller and management interfaces such as antivirus, role based access for secure management, authentication and authorization, audit logging, and directory services.

 

So, if your storage vendor is planning to consolidate their storage platforms without a plan for migration, or you are struggling with a number of legacy storage platforms and would like to upgrade to the latest enterprise capabilities without ripping and replacing your storage estate, please consider using Hitachi’s VSP storage platform to help you with the migration and consolidation. Our VSP systems can come without any internal storage attached in the event that you already have enough capacity in your legacy storage and you want to virtualize and extend the useful life of your current storage, or you want to use it as a sling box to move data across storage systems. However, the biggest benefit would be to leverage all the capabilities that are available in the VSP/SVOS RF

Forbes believes it is an imperative for CIOs to view cloud computing as a critical element of their competitiveness. Cloud-based spending will reach 60% of all IT infrastructure and 60-70% of all software, services, and technology spending by 2020. In 2019, CIOs will have to optimize applications of the newest cloud technology in response to their requirements.

rean.png

Forbes notes that a full transition to the cloud has proved more challenging than anticipated and many companies will use hybrid cloud solutions to transition to the cloud at their own pace and at a lower risk and cost. This will be a blend of private and public hyperscale clouds like AWS, Azure, and Google Cloud Platform. CIOs will rely upon migration assessment and planning activities to identify an optimal allocation of workloads across public and private cloud environments. Private clouds are not simply existing data centers running virtualized, legacy workloads. They require highly modernized digital application and service environments running on true cloud platforms like Hitachi Enterprise Cloud. Hybrid clouds must bond together the two clouds through fundamental technology, which will enable the transfer of data and applications.


While public cloud vendors offer infrastructure as a service (IaaS,) which delivers compute, storage and network resources in a self-service, highly automated fashion and platform as a service (PaaS), such services do not completely eliminate the need for IT operations management. Customers still need expertise to choose the right service elements and to configure them appropriately, and they retain responsibility for the guest OS, middleware and applications that run on their IaaS compute instances. Public cloud also introduces new challenges in governance, financial management and integration.


Customers look to third parties for transitioning to public cloud, due to lack of expertise or staffing. Engagements may be on a short-term tactical basis or as part of a long-term managed service. As a result, an ecosystem of managed and professional service providers has developed to provide Managed Service Providers (MSP) for Public Cloud. This business has grown to the extent that Gartner has developed a Magic Quadrant for MSPs that offer managed services and professional services related to infrastructure and platform operations for one or more hyperscale integrated IaaS+PaaS providers. The term “hyperscale” is used by Gartner to refer to Amazon Web Services, Microsoft Azure, and Google Cloud Platform.


One of the vendors that has been on the Gartner Magic Quadrant for Public Cloud Infrastructure Managed Service Providers, Worldwide since its first release in March 2017, is REAN Cloud. As we recently announced, Hitachi Vantara has finalized the acquisition of REAN Cloud to directly complement the strength of our Hitachi Enterprise Cloud (HEC) and on-premises deployments. REAN Cloud is a global cloud systems integrator, managed services provider and solutions developer of cloud-native applications across big data, machine learning and emerging internet of things (IoT) spaces. Through this acquisition, Hitachi Vantara gains critical capabilities and industry-leading expertise in cloud migration and modernization, instantly elevating its cloud offering portfolio and managed services capabilities.

 

REAN Cloud has expertise working with the hyperscale public clouds. They are a Premier Consulting Partner in the Amazon Web Services (AWS) Partner Network (APN) and a Microsoft Azure Silver Partner. REAN Cloud offers managed services and solutions for hyperscale-integrated IaaS and PaaS providers and is one the few systems integrators capable of supporting the entire cloud services life cycle. Backed by extensive security DNA and deep compliance IP and expertise, REAN Cloud specializes in helping enterprise customers that operate in highly regulated environments – Financial Services, Healthcare/Life Sciences, Education and the Public Sector – accelerate their cloud investments while extracting maximum value from use of the cloud itself.


Last year REAN Cloud acquired 47Lining to provide deep capabilities in cloud-based analytics and machine learning that expands Hitachi Vantara’s ability to maximize data-driven value for vertical IoT solutions. This April, 47Lining, announced its Amazon Web Services (AWS) Industrial Time Series Data Connector Quick Start . The Connector Quick Start allows companies to quickly and easily synchronize their industrial time series data to AWS so they can perform advanced predictive and historic analytics using the full breadth of AWS big data and partner services.


The REAN Cloud team will be joining Hitachi Vantara’s Global Services organization, which is led by Bobby Soni, chief solutions and services officer. In July when we announced our intent to acquire REAN Cloud, Bobby posted this blog which gives his view on how REAN Cloud will be integrated into our roadmap to offer our ecosystem of partners and customers a secure and unified multi-cloud platform. What really excites Bobby about REAN Cloud is its people as we can see in this quote from his blog post:

 

The REAN Cloud team is highly talented, passionate about cloud and big data engineering, and are clearly at the top of their fields. Data scientists, DevOps engineers, big data consultants, cloud architects, AppDev engineers, and many more – all of them smart and collaborative. We are all thrilled to welcome them to our own team of talented professionals. Together, we will continue to push the envelope by creating unique and insightful analytics and IoT solutions in collaboration with our partners and customers.”

 

Hitachi Vantara's cloud solutions offer unified cloud management for workloads across public, private and hybrid cloud environments. By combining REAN Cloud's expertise in public cloud and intelligent data governance with Hitachi Vantara’s robust global delivery ecosystem, Hitachi Vantara will now be better positioned to meet the growing infrastructure and analytics-based needs of its customers as they transition to the cloud.

democratization.png

Since the acquisition of EMC by Dell in 2016, the Dell/EMC product line has been fragmented, with VNX, VNXe, EqualLogic, Compellent, Isilon, SC Series, XtremIO, Unity, VMAX, VMAX3, PowerMax, etc. In May, CRN reported that Dell/EMC ‘s “new storage strategy will have Dell engineering teams focusing squarely on one primary storage product line for each market segment: high end, midrange, low-end, and a separate product for the unstructured file and object storage market.”

In a May 21 interview with CRN Dell Vice Chairman Jeff Clarke unveiled the midrange roadmap, which was to breakdown silos and consolidate four midrange products and concentrate on one. That was the extent of the “Roadmap”. This was critiqued by HPE’s Phil Davis who wondered which of these architectures would survive, and what this will likely mean for customers in this CRN article. Jeff Clarke responded in another CRN article saying "Despite what may be being said by others, our products on the roadmap today are supported throughout their lifetime period and we will provide a non-disruptive migration from the old product to the new product," Pure Storage responded to this with a blog by Ken Steinhardt which questioned how the many current products would be supported and how “non-disruptively migration would be accomplished.

In Jeff Clarke’s response to his critics in the CRN article above, he said:

“When I talked to customers and said, 'Would you rather buy four midrange products or one, and the right one, with all of our best IP?' The answer is overwhelmingly positive to the degree of 100 percent that, 'We'd prefer to buy one product that's the right product with all of the Dell EMC technology and IP in it,'"

With all this back and forth generating a lot of internet chatter, Hitachi Vantara has been sitting on the sidelines watching this debate. Having one product for midrange is a no brainer, but it has taken Dell EMC several years to realize the need, and it will take several more years for them to implement this for the midrange. Having one product for midrange does not go far enough. Hitachi Vantara believes that even more benefits can be realized with one platform not only for the midrange but for all three market segments together, high-end and mainframe as well as midrange. In their 2017 Magic Quadrant for general purpose disk arrays, Gartner acknowledged that sharing a common architecture and management tools from the smallest VSP G200 to the flagship VSP G1500 preserves customer investments in policies and procedures, and leverages ecosystem-related investments. Dell EMC has acquired so many different storage products that it will be very difficult for them to integrate them without a major disruption to existing customers. Dell EMC is not alone. All major storage vendors, including IBM and HPE, have multiple storage platforms that cannot be integrated for operational efficiency. This creates complexity, limits agility, and the risk of stranding some products without an upgrade path or abandoning some market segments completely.

Unlike other storage vendors who have had to acquire a good portion of their storage architectures and technologies and are struggling to integrate them, Hitachi Vantara, has had the benefit of a strong R&D parent with over 4,000 patents in storage technology, developing our own unique architecture. The unique architecture, in our Virtual Storage Platform (VSP), has enabled us to provide not only a common storage platform for our midrange customers, but also for our high-end and mainframe customers. The VSP platform is powered by the Storage Virtualization Operating System (SVOS) which enables common management and capabilities across the VSP family of open-system, cloud, containerization, and mainframe solutions. Even when you need to add different storage personalities for the unstructured file and object storage market, Hitachi’s strategy is designed to let you consolidate data on a single platform. Object (HCP) and file (HNAS) can be deployed as gateways that use the same VSP / SVOS on the backend.

VSP Family.png

The physical packaging of the different models is optimized for their use case. For instance, the hybrid flash (VSP G series) and all flash VSP models (VSP F series) for the midrange are packaged in a cost optimized 2U controller while the controller for the VSP for mainframes is packaged in a 10U controller chassis that leverages blades for easy upgrades of compute capabilities and connectivity for massive scalability as well as mainframe FICON connections. However, the SVOS operating system remains the same. What this means is that customers with a midrange VSP can utilize functions like Global-active device for zero RTO (Recovery Time Objective) and zero RPO (Recovery Point objective) metro clustering and add remote replication for 3 data center protection which other vendors can only provide in their high-end systems, if at all. Support for containers to allow DevOps to accelerate cloud-native application development, virtualization of externally attached storage for non-disruptive device migrations, and a 100% data availability guarantee are available across our VSP/SVOS platforms. Our latest version of our storage virtualization operating system SVOS RF which was released last year enables us to build the SVOS microcode so that a “single package” could be used on multiple VSP’s.

From a value standpoint, having a common SVOS platform avoids the technical debt incurred by having multiple platforms, stranded storage on different systems, management & service complexity, and the overhead of managing different code upgrade cycles. The fact that data services are common across all offerings enables the “democratization” of storage services. A midrange user now has access to the same, super-powerful features as the biggest banks. Or, as a large company, you can deploy the same super-powerful features in remote, edge offices as you do in the core. This helps for data protection because you can easily use a single offering to replicate data back to the core. It also helps with best practices because you can use the same run books for deploying storage OR use the same Hitachi Automation Director policies everywhere. This makes global automation easier to roll out and helps de-risk operations anywhere in the world for better uptime and superior customer experiences.

Hitachi Vantara is the only storage vendor that provides all of the Hitachi technology and IP for not only our high end and mainframe customers, but also for our midrange customers as well. We democratize storage services so that all can enjoy the benefits of our proven technology. You don’t have to wait two or three years to see a roadmap evolve. It is here today with Hitachi Vantara. For information on Hitachi Vantara’s Storage portfolio visit our website at this link

GUIDE SHARE EUROPE is holding their 2018 GSE UK Conferencewhich will take place at Whittlebury Hall on November 5th, 6th and 7th 2018, in Whittlebury, UK. This conference will provide 3 days of intensive education across a broad range of mainframe topics.

 

whittlebury.png

 

GUIDE SHARE EUROPE, GSE for short is a European user group for IBM mainframe hardware and software. Guide was an international user group for IBM, founded in 1956. It grew to be the largest IBM user group until the 1990’s. In 1994 GUIDE in Europe merged with SHARE, another large IBM use group and became GUIDE SHARE Europe. In the US, GUIDE ceased to exist and many of its activities and projects were taken over by SHARE. GSE like Share is primarily focused on IBM Mainframes and has more than 1300 organizations in Europe, which shows that mainframes still have a large presence there. GSE assists its members in exchanging experience and information, assessing developments in information technology, and providing them with opportunities to further develop and influence IBM's product policy.

 

SHARE is a volunteer-run user group for IBM mainframe computers that was founded in 1955 and is still active today providing, education, professional networking, and industry influence on the direction of mainframe development. SHARE member say that SHARE is not an acronym, it is what they do. SHARE and GUIDE were the precursors of the open source communities that we have today.

 

The mainframe market is alive and well and may be on the verge of a renaissance in the coming IoT age. We have all seen the staggering projections for 30+ billion new internet connected devices and a global market value of $7.1 trillion by 2020. That is almost 8 times the estimated 4 billion smartphones, tablets, and notebooks connected today. That translates into a staggering amount of additional new transactions and data, which means compute and data access cycles, as well as storage. That many new devices connected to the internet also opens up many more security exposures.

 

These are areas where the mainframes excel with their unique architecture of central processing units (CPUs) and channel processors that provide an independent data and control path between I/O devices and memory. With z/OS, the mainframe operating system, is a share everything runtime environment that gets work done by dividing it into pieces and giving portions of the job to various system components and subsystems that function independently. Security, scalability, and reliability are the key criterions that differentiate the mainframe; and are the main reasons why mainframes are still in use today especially in high transaction high security environments like core banking. These same capabilities will be required by the backend systems that support IoT.

 

Hitachi Vantara is one of the few storage vendors that support mainframes with its scalable VSP enterprise systems. Our own Ros Schulman will be presenting two sessions:

 

Virtual Storage Platform Update - Latest and Greatest: November 5 at 13:00.

In this session Ros will discuss the latest updates to the Hitachi Vantara G1500 and F1500 Storage subsystems. This update will include hardware, software, and solutions as well as options for moving data to the cloud.

 

Data Resilience and Customer Solutions for the Mainframe in the IoT Age: November 5 at 1530

Ros will discuss effects of the IoT age on mainframes, the importance of data resilience and analytics and Hitachi Vantara IoT solutions available today for mainframe environments, and where we may be headed in the future.

 

For other sessions, please see the link to this Agenda

At NEXT 2018 this last September, John Mcgee, our VP of Portfolio Marketing hosted a session on What’s NEXT: Innovation at Hitachi. In this session he explained how Hitachi thinks about innovation, how Hitachi is bringing new solutions to market, and introduced some of the things that we are looking at further down the road. One of the areas he emphasized was the need for data driven insights.

John McGee.png

John pointed out that every application is becoming an analytics app. How do you get more insights to drive business outcomes? Machine learning apps are getting updated and refreshed. Every customer interaction is informed by the latest data and the latest model. It’s not like the old days when you develop an app and put it into production, now it is about continual development and learning. Data is no longer processed in the back room by data scientists working on batch data. The data must be streaming in real time, where we inject it with meta data, so we know the context of the data and can curate it to provide the right data to the right consumer for business purposes.

FPGA.png

To assist with data driven insights, one of the areas for future development is in the area of Field Programable Gate Arrays (FPGAs). FPGAs are semiconductor devices that are based around a matrix of configurable logic blocks (CLBs) connected via programmable interconnects. FPGAs can be reprogrammed to desired application or functionality requirements after manufacturing. More importantly FPGAs are wired just like electronic circuits, so you can have truly parallel circuits. Unlike micro-processors that process code sequentially or jumps from one piece of code to another to simulate parallelism, FPGAs can process multiple code in multiple parallel circuits, which is ideal for accelerating tasks like graphics processing or machine learning at a much lower cost than micro-processors. If you are not familiar with the benefits of FPGAs, take a few minutes to view this YouTube Video

Hitachi has been working with FPGA technology for some time with thousands of man hours invested in research and development, producing over 90 patents. The most visible use of FPGA’s in Hitachi products is our high-performance Hitachi NAS which enables us to run 4 parallel processing streams where traditional NAS controllers run one. What is exciting will be the use of FPGAs in accelerating a variety of analytics applications with Pentaho analytics being a first use case and Proof of Concept. Pentaho engineers have demonstrated 10 to 100 time faster analytics with much less space, much less resources, and at a fraction of the cost to deploy.

 

The parallelism of FPGAs makes them very fast, but they have been very specific to use case and workload. So, what the researcher are working on is a software defined FPGA accelerator that can use a common FPGA platform on which we can develop algorithms that are much more transferable across workloads. The benefit will be the acceleration of insights on many analytic opportunities, many different application types, and brings things out faster to market. In this way we hope to crunch those massive data repositories and deliver faster business outcomes and solve social innovation problems. It also means that as we see data gravity pull more compute to the edge, we can vastly accelerate what we can do in edge devices with less physical hardware because of the massive compute and focused resources that we can apply with FPGAs.

Software Defined FPGA.png

Other topics covered in this presentation were Edge to Cloud data services, Neural Network Storage Research, Digital Twins, Machine-learning Model Management, and Optimizing Operations  with Video Analytics.

 

If you missed this presentation at our NEXT2018 event you can see a video of this presentation at our NEXT website which contains most all the presentations at this event. This particular presentation is under General session Day 2.

UCP Technology.png

Among the many announcements that were made at Hitachi Vantara’s NEXT 2018 event last month in San Diego, was the announcement of an enhanced converged and hyperconverged portfolio to help customers modernize their data centers as part of their digital transformation strategies. Hitachi Vantara is committed to providing customers of our IT infrastructure products with the latest technology innovations to address their evolving business and data requirements. There was a lot that was covered in this announcement, so t would like to go into more detail on four of the key innovations that I feel are more significant. These Innovations include: DS7000 Scalable Servers, NVIDIA Tesla GPUs, All NVMe, and 3D XPoint storage memory.

 

DS7000 Scalable Servers

 

Hitachi Advanced Server DS7000 Series of Scalable Servers are built with a unique modular architecture which can be configured and scaled to meet the needs of a wide variety of application workloads. Each module supports a pair of Intel Xeon Scalable processors, with up to 56 cores and 3TB of memory per module. An 8 socket DS7000 server (DS7080) can support up to 224 cores and 12TB of memory. Each model can be smoothly upgraded to the next, preserving your investment in hardware and software as you grow, and compute modules can be individually configured to support a wide variety of compute and storage options. The DS7000 server family is available in Converged and Rack Scale configurations and is the ideal platform to deliver the high availability and scalability needed for Hitachi’s solutions for business-critical applications. Hitachi Solution for the SAP HANA Platform makes full use of the huge memory capacity of the DS7000 to deliver real-world business benefits from data analytics. Hitachi Solution for Databases relies on the powerful performance and massive I/Ocapacity of DS7000 to meet demanding service level agreements (SLAs) and rapidly changing workloads, allowing businesses to thrive and grow

 

Hitachi Advanced Server DS225 with NVIDIA Tesla GPUs

 

GPUs which were first developed for the gaming industry can now play an important role in the modern data center. Hitachi Advanced Server DS225 with NVIDIA Tesla GPUs is architected to accelerate advanced analytics, Machine Learning, CAD/CAE, and specialized graphics.

 

GPUs and CPUs process tasks in different ways: A typical CPU is optimized for sequential serial processing and is designed to maximize the performance of a single task within a job. GPUs, on the other hand, use a massively parallel architecture aimed at handling multiple functions at the same time. As a result, GPUs are 50 to 100 times faster than CPUs in tasks that require multiple parallel processes such as machine learning and big data analysis. The following Youtube Myth buster video is a fun way of showing the difference between CPU and GPU processing.

 

 

According to NVIDIA, data scientists and researchers can use their NVIDIA® Tesla® GPUs to parse petabytes of data, orders of magnitude faster than they could by using traditional CPUs, in applications ranging from energy exploration to deep learning, enabling them to tackle challenges that were once thought impossible. Tesla accelerators also deliver the horsepower needed to run bigger simulations faster than ever before and deliver the highest performance and user density for virtual desktops and workstations.

 

The DS225 with NVIDIA can be combined with the DS7000 server in the   Hitachi Unified Compute Platform CI (UCP CI) converged systems and Hitachi Unified Compute Platform RS (UCP RS) rack-scale systems to address different environments, use cases and applications.

 

All NVMe and 3D XPOINT

 

Hitachi Unified Compute Platform HC (UCP HC) hyperconverged infrastructure appliance series will include a new all-flash NVMe UCP HC V124N system providing high storage density and capacity in a low-latency 1U form factor housing up to 12 NVMe SSDs (up to four ultra-fast Intel 3D XPOINT-based Intel Optane SSDs for cache and eight NVMe SSDs for capacity) delivering a total raw capacity of up to 80TB. It is designed to address the needs of customers requiring higher IOPS and ultra-low latency for applications such as relational databases, real-time analytics, stock trading, fraud detection and  transaction processing (OLTP). The all-NVMe hyperconverged system will offer customers dramatically improved performance and efficiencies with a higher ROI by delivering hundreds of thousands of IOPS at sub-millisecond response time, for a 3X increase in IOPS performance and 4X lower latency compared to the prior all-flash UCP HC system. Hitachi and Intel engineering teams collaborated to run joint tests over many months to determine and optimize the performance results for real-world scenarios. For more information about the all NVMe hyperconverged solution and the test results,  you can read my colleague Dinesh Singh’s blog here.

 

NVMe (non-volatile memory express) is a host controller interface and storage protocol created to accelerate the transfer of data between computers and solid-state drives (SSDs) over a computer's high-speed Peripheral Component Interconnect Express (PCIe) bus. (You can read my previous blog post on NVMe and Enterprise storage arrays via this link)

 

While NVMe NAND SSDs have been available on UCP hyperconverged systems for some time, this latest enhancement introduces a new solid-state device which is a major improvement over NAND SSD technology. Intel’s 3D XPOINT technology is a new type of persistent solid-state memory which sits between the performance of NAND SSD and DRAM. 3D XPoint has about one thousandth the latency of NAND flash (or about ten times the latency of DRAM), and ten times the density of DRAM. Since its cost is higher than NAND SSD but lower than DRAM it can be used as a cache between NAND SSD and DRAM to boost performance. Here is another YouTube Video that does a great job of describing the 3D XPoint technology.

 

 

We will be showcasing all-NVMe hyperconverged system Unified Compute Platform HC at VMworld Barcelona in a few weeks. Please visit Hitachi booth #P311for a live demo and see how to beat latency.

 

 

Summary

 

These new innovative technology options provide a flexible, low-risk path to modernizing data center infrastructure for a broader set of enterprise applications. New mix-and-match server options increase server flexibility and density to address different environments, use cases and applications.

NEXT Winners.png

Hitachi Vantara’s NEXT 2018 event for customers and partners concluded last week. We show cased over 50 discrete solutions in our showcase, made 9 unique announcements and revealed some of the work that we are doing in our Research labs and Centers for Social Innovation. More importantly we had over 30 customer advocates share their digital Transformation successes with Hitachi Vantara.

 

NEXT DX.png

 

There were many examples of customers realizing their digital transformation with Hitachi and Hitachi Vantara solutions and support. Some of the customer and partner who shared their digital transformation experiences were;

NEXT Customer Advocates.png

NEXT 2018 attendees and the general public were encouraged to vote for their favorite story for the Hitachi Transformation Awards. The award winners included:

 

NASA Johnson Space Center for Enterprise transformation, partnered with Hitachi Vantara to design and implement a unique, cloud based, long-life digital repository for mission imagery to enable instant access by authorized users around the world.

 

Smart Payments by NETs for Social Innovation, partnered with Hitachi to co-create an advanced next-generation payments solution using finger vein biometric technology.

 

State of Andhra Pradesh(India) for Excellence in IoT, forproviding agile and efficient Real Time Governance (RTG) public services to their 50 million citizens with the help of Hitachi to analyze big data sets gathered from various sources to make insightful decisions.

 

Honorable mentions included:

 

Deluxe who transformed from check printing to a financial services and small business marketing company using a world leading managed-services cloud ecosystem.

 

City of Las Vegas who is working with Hitachi Vantara to create a new, dynamic and connected city to fuel innovation, efficiency, and new ways of government to better the experience for residents, businesses and visitors.

 

          PT. Angkasa Pura II (Persero),one of the state-owned enterprises engaged in airport services and airport-related

          services in Indonesia, using video analytics to improve operational management, increase awareness and provide

           an excellent customer experience for over 100 million passengers annually.         

 

We also announced two Collaboration efforts with Health City and Arrow Electronics

 

Hitachi Vantara announces collaboration with Health City. Health City is a cluster-led, economic development organization driven to take health innovation to new levels for the city of Edmonton, Alberta, Canada. Hitachi and Health City will establish a co-creation relationship focused on innovative technology solutions that facilitate opportunities for health innovation in Alberta. This collaboration brings together Hitachi’s leading data and information management technologies with Health City’s focus on regional economic development, while advancing health and social well-being. This collaboration is designed to scale over time to allow the development of repeatable solutions for any community.

 

Hitachi Vantara and Arrow Electronics announced the next phase of its successful 20-year affiliation. Together, the companies will expand their combined efforts beyond high-end storage to include the development and sales of new end-to-end market-ready solutions powered by proven infrastructure, analytics, internet of things (IoT) applications, vertical solutions, network and connectivity, gateways, sensors and edge technologies from Hitachi Vantara and Arrow, as well as other ecosystem partners. Earlier this month, Hitachi Vantara announced that it joined Arrow as a technology adviser and partner in the Colorado Smart Cities Alliance (see recent press release). Hitachi Vantara and Arrow will help Colorado municipalities utilize data-driven insights derived from IoT, big data, advanced analytics and video intelligence technologies and solutions to fuel innovation, solve local challenges and improve smart city development.

 

 

The theme of this NEXT 2018 Event was Your Data,Your Innovation. I could write a separate blog post on how each of these companies have created innovation with their data and I might sprinkle some of these stories in future posts. In the mean time you can can download the Videos and presentations of this event at HitachiNEXT.com

 

I would also like to thank our sponsors, not only for helping to make this event a success, but more importantly working with us to help our customers co-create and innovate.

 

Diamond Sponsor:      Intel

Platinum Sponsor:      AWS, Arrow, DXC.technology, Google Cloud, VMware

Gold Sponsors:            Atos, CISCO, Veritas

Silver Sponsors:          Aptare, Broadcom, Commvault, Extreme, Flexential, Red Hat, SUSE, TechData, 

                                    Verizon, Virtual Instruments, XORUX

Virtual Sponsors:        Cognizant, Juniper Networks, Mesosphere

NEXT 2018.png

 

At Hitachi Vantara’s NEXT 2018 event, this week in San Diego, we were fortunate to have Malcolm Gladwell as a keynote speaker. Malcolm Gladwell is a writer for The New Yorker and has published five books that have been on the Times bestselling list: Tipping Point, Blink, Outliers, What the Dog Saw, and David and Goliath. Malcolm’s books and articles often provide unexpected insights which helps us understand the events around us and enable us to make better decisions.  

 

In the general session our CEO Brian Householder did an interview with Malcolm. Some of the ideas that came out were: A model is only as good as the data that goes into it which he illustrated with the use of standardized student testing to measure the quality of teachers. Another was to focus on the core issue and not to panic when the circumstances changes. The example here was the music industry where the business shifted from recordings to  and streaming music. The music industry was panicked over the loss of revenue, but today the music industry is making more money than ever due to live performances which are promoted by their  music. I could see a similar transition in our industry where cloud was a threat to the IT vendors, but today the revenues are increasing for IT vendors due to software and services which makes it easier for their customers to develop applications and generate information. 

 

Later I had the opportunity to moderate a Q&A session with Malcolm and a group of VIP customers. Here we started with the paradigm of puzzles versus mysteries. While Malcolm is known for his best sellers, there are a lot of ideas that are created in his New Yorker articles that are creating even more interest today. In 2007 he wrote an article “Open Secrets “, in which he raised the paradigm of puzzles versus mysteries.

 

For example the whereabouts of Osama Bin Laden was a Puzzle. This was eventually solved and taken care of. Puzzles can be solved by gathering more data. There is an answer. On the other hand, what happens to Iraq after the fall of Saddam Hussein was a mystery. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much. A mystery is a problem caused by an excess of information--and can only be solved by trying to make sense in a more sophisticated way of what is already known. Today we live in an age of information overload and all professions must manage the transition from puzzle solving to mystery solving.

 

While the direction of the Q&A did not allow him to go deeper into this topic, he has talked about this in other interviews. In an interview in inc.com. He said that most businesses and industries are built for solving puzzles, not mysteries. With greater access to data comes greater responsibility. Mysteries require a shift in thinking that most industries simply are not organized or prepared to handle. Complex mysteries require a different kind of thinking and analytic skills. You need to decide whether the problem you are solving is a puzzle or a mystery. If you think you are solving a puzzle and collect more and more data, the overload of data might bury the key nuggets that could help you solve the mystery. Data that is used to solve a puzzle must be looked at differently when you are solving a mystery. One of the biggest mistakes that businesses make is treating all data equally, by giving all data equal weight.

 

“This idea gets back to Nate Silver's The Signal and the Noise concept. Some data will tell you precisely what you need to know and help you solve the mystery you're after, but most data is noise, distracting you from the answers you seek. Today, the trick is not analyzing the data so much as understanding which data represents the signal you pay attention to versus the data that is the noise that you must ignore. We have to start ranking data in terms of its value.”

 

The transition from puzzles to mysteries should resonate with most CIOs in regards to their data. Data bases were great at solving puzzles, but the complex nature of today’s business is more of a mystery that requires big data and analytics. In order to be competitive, companies are shifting from data generating to data powered organizations and big data systems are becoming the center of gravity in terms of storage access and operations. Data curation is needed to understand the meaning of the data as well as the technologies that are applied to the data so that data engineers can move and transform the essential data that data consumers need. Hitachi Content Intelligence and Pentaho Data Integration are key tools for searching, classifying, curating, enriching, and analyzing the data to understand what you have and how it can be used to solve mysteries.

Don’t Blink or you’ll miss What the Dog Saw. Come to NEXT 2018, the Premier Event for the Digital Revolution, sponsored by Hitachi Vantara in beautiful San Diego, Sept. 25-27. Catch the Outliers and what may be the Tipping Point for your successful transformation with your data, your innovation.

 

Malcom.png

 

By now you may have recognized the titles of a few of the bestselling books of Malcolm Gladwell, one of Time magazines 100 most influential people and one of the Foreign Policy’s Top Global Thinkers. Gladwell is a story teller who makes you see things in different ways. He is an assumption challenger. He has explored how ideas spread in The Tipping Point, decision making in Blink, the roots of success in Outliers, and in What the Dog Saw, he has compiled several interesting psychological and sociological topics ranging from people who are very good at what they do but are not necessarily well known to intelligence failures and the fall of Enron.

 

Malcolm Gladwell also authors, Revisionist History, a podcast, produced through Panoply Media. It began in 2016, and has aired 3 10-episode seasons. Revisionist History is Malcolm Gladwell's journey through the overlooked and the misunderstood. Every episode re-examines something from the past—an event, a person, an idea, even a song—and asks whether we got it right the first time. Being a veteran of the Vietnam War, I was particularly interested in Saigon, 1965, If you are interested in history you would be interested in this podcast.

 

Malcolm Gladwell will be addressing the topic: What Disruptors Have in Common on Wednesday morning Sept 26 at NEXT 2018. If you can’t make it to NEXT 2018 you can watch the livestream of Malcolm @Gladwell’s keynote on http://htchivantara.is/NEXT

 

Don't miss this event and a chance to hear Malcolm Gladwell's insights into the changes facing our industry!

Big data refers to the use of data sets that are so big and complex that traditional data processing infrastructure and application software are challenged to deal with them. Big data is associated with the coming of the digital age where unstructured data begins to outpace the growth of structured data. Initially the characteristics of big data was associated with volume, variety, and velocity. Later the concepts of veracity and value were added as enterprises sought to find the ROI in the capture and storing of big data. Big Data systems are becoming the center of gravity in terms of storage, access, and operations; and businesses will look to build a global data fabric that will give comprehensive access to data from many sources and computation for truly multi-tenant systems. The following chart is a composite of IDC’s 2018 Worldwide Enterprise Storage systems Market Forecast 2018-2022and Volume of big data in data center storage worldwide from 2015 to 2021. This composite shows that Big Data will eventually be more than half of the capacity of enterprise data by 2021.

 

Big Data Growth.png

 

Unfortunately, the deployment of large data hubs, data warehouses, data lakes, ERP, Salesforce, and Hadoop, has resulted in more data silos that are not easily understood, related, or shared. Deployments of large data hubs over the last 25 years (e.g., data warehouses, master data management, data lakes, Salesforce and Hadoop) have resulted in more data silos that are not easily understood, related, or shared. In the May-June issue of the Harvard Business Review,  Leandro Dalle and Thomas Davenport, published an article “ What’s your Data Strategy”, in which they claimed that less than 50% of an organization’s structured data is actively used in making decisions and less than 1% of its unstructured data is analyzed or used at all. While the ability to manage and gain value from the increasing flood of data is more critical than ever, most organizations are falling behind. I contend that much of this is not due to the lack of interest or need but is due to the difficulty of accessing the silos of data. I believe we need to add another “V” word in association with big data. That word is virtualization.

 

In IT there have been two opposite approaches to virtualization. Server virtualization where you make one physical server look like multiple servers, and storage virtualization where you make multiple physical storage units look like a single storage system – a pool of available storage capacity that can be managed and enhanced by a central control unit. The virtualization of big data is more like storage virtualization, where multiple data silos can be managed and accessed as one pool of data. The virtualization of data is done through the use of meta data, which enables diverse data to be stored and managed as objects.

 

Object storage can handle the volume challenge. It is essentially boundless since it is a flat file and is not bound by directories. Hitachi’s HCP (Hitachi Content Platform) can scale the volume of data across internal and external storage systems, and from edge to cloud. HCP meta data management enables it to store a variety of unstructured data. Hitachi Vantara’s object store was designed for immutability and compliance and has added Hitachi Content Intelligenceto ensure the veracityof data that it stores.  HCP with Hitachi Content Intelligence provides a fully federated content and metadata search across all data assets, with the ability to classify, curate, enrich and analyze the data you have. Hitachi Vantara’s Hitachi Content Platform and Hitachi Content Intelligence provide the intelligent generation and curation of meta data to break down those silos of unstructured data. Pentaho with its Pentaho Data Integration (PDI) provides a similar capability to breakdown the silos of structured data.  In this way we can virtualize the silos of data for easier access and analysis and transform these separate data hubs into a data fabric.

 

Earlier, we mentioned that Velocity was one of the attributes of big data. That referred to the velocity at which unstructured data was being generated. It did not refer to the access speed of object storage systems. Since object storage systems use internet protocol, it cannot process data as fast as directly attached file or block systems. As a result, large analytic systems like Hadoop would ETL the data into a file system for processing. For years Hadoop has been the go to data storage and processing platform for large enterprises. As Hadoop has solved this problem of storing and processing massive amounts of data, it has created a new problem. Storing data in Hadoop is expensive and fault tolerance comes from 3x data redundancy. Storing this data indefinitely is expensive but this data is still valuable, so customers do not want to throw it away. HCP with its lower cost object storage options and 2x data redundancy, can solve the problem of storing and protecting massive amounts of data. But what about the lack of processing speed? This is where virtualization can also help.

 

Hitachi Content Platform (HCP) has partnered with Alluxio to utilize their memory-speed virtual distributed file system to deliver a certified solution that simplifies the challenges of connecting big data applications, like Hadoop, to Hitachi Content Platform to reduce storage costs, and provide high-performance and simplified access to data. Alluxio lies between compute and storage and is Hadoop and object storage compatible. Alluxio intelligently caches only required blocks of data, not the entire file which provides fast local access to frequently used data without maintaining a permanent copy of data. Existing data analytics applications, such as Hive, HBASE, and Spark SQL, can run on Alluxio without any code change and store data to HCP and access data with high performance memory-speed access

 

This is similar to the what HCP provides as a virtual bottomless file system for NFS and SMB filers, where files are accessed through a local Hitachi Data Ingestor which acts as a caching device to provide remote users and applications with seemingly endless storage. As the local file storage fills up, older data is stubbed out to an HCP system over S3. Cloud storage gateways like HDI, HCP Anywhere edge file servers or 3rd party tools’, act as a cloud gateway connecting remote sites to the data center without application restructuring or changes in the way users get their data. This eases the migration from traditional NAS-to-cloud-based file services.

 

Now is the time to add Virtualization to the Big Data “Vs” of Volume, Variety, Velocity, Veracity, Value.

Aliyun2.png

Amazon’s cloud business grew nearly 48.9$ in the second quarter of 2018 generating $6.11 billion in revenue which contributed 11.5% to Amazon’s total revenues in that period. Their cloud business also was one of the main contributors to Amazon’s record $2.5 billion in profit. This is a strong sign that the cloud business is growing and adding value to organizations.

 

In the early days of cloud there were concerns about trusting your data to the cloud. There were concerns about security, reliability and availability which are core to any IT operation. There were also disasters where cloud companies like Nirvanix, one of the pioneers of cloud-based storage went out of business and gave customers only two weeks to retrieve and remove their data. Several telco’s started, then withdrew from the cloud provider business. Amazon’s cloud business was introduced in 2002 and did not turn a profit until 2015. However, cloud technologies and storage technologies have moved us past those concerns and enterprises are moving more of their data to the cloud. In 1Q 2018, IDC reported that the revenues generated by original design manufacturers (ODM) who sell storage directly to cloud hyperscale data centers increased 80.4% year over year to $3.1 billion and was 23.9% of total enterprise storage investments during that quarter.

 

One of the cloud technologies that has increased trust in the cloud is S3. S3 (Simple Storage Services) is a cloud computing web service which provides object storage through web services interfaces like REST using HTTP commands that enable data to travel across the internet. The S3 API provides the capability to store, retrieve, list and delete objects in S3. S3 services include features that have increased the trust and usability of cloud storage such as multi-tenancy, security& policy, atomic updates, life cycle management, logging, notifications, replication, encryption, and billing. S3 was introduced by Amazon in 2007 and has been adopted by most major cloud providers.

 

The major storage technology that increased trust in the cloud is object storage with an S3 interface. Hitachi’s object storage, Hitachi Content Platform (HCP) enables data to be tiered to the cloud over S3. Tiering enables an enterprise’s data center to move data that is less active to lower cost storage capacity in the cloud while maintaining control of the object storage from the data center. HCP always maintains at least two copies of the data objects and eliminates the need for backups. The copies can be dispersed over a geographic area with geo distributed erasure coding for availability. HCP also provides the flexibility to migrate data across the different cloud platforms or, in the event of another Nirvanix type failure, continue operations while recovering the data.  Migrating between clouds involves setting up a new account in another cloud and crypto shredding the data in the old. Operations continues on the second copy and no movement has to be done between the clouds, which would be very expensive since cloud storage involves access charges.

 

Since the S3 interface is being broadly adopted by cloud providers, it is relatively easy to interface HCP to most clouds. We have connected to many of the major cloud venders in our Center of Excellence. Jeff Lundberg posted a blog on how we successful tested data tiering from HCP to the Alibaba Cloud (Aliyun). This connection to Aliyun is exciting for us since Aliyun is the fastest growing cloud provider.

 

While Aliyun is only about one tenth the size of Amazon and is not yet profitable, it is growing faster than Amazon. According to a recent Synergy research report Ali Baba surpassed IBM for the 4th ranking in worldwide cloud infrastructure, behind Microsoft Azure, Google Cloud, and of course Amazon AWS. Ali Baba is the largest cloud computing company in China,[1] and operates in 18 data center regions and 42 availability zones around the globe. One is located near us in San Mateo, California. As of June 2017, Alibaba Cloud is placed in the Visionaries' quadrant of Gartner's Magic Quadrant for Cloud Infrastructure as a Service, Worldwide.

 

HCP’s unique object storage service plus S3 tiering provides organizations with a secure, efficient and cost-effective way to leverage cloud services from multiple cloud vendors while retaining control within their private data centers.

In the last few days this week we have seen a massive sell off in crypto currencies. Bitcoin lost $1000 in 24 hours. Other currencies like ethereum, ripple, bitcoin cash and EOS, all dropped by around 20 per cent. As of this Thursday, one Bitcoin is worth $6,486.45 USD. This is down from nearly $20,000 in December 2017.

 

Bitcoin meltdown.png

One reason for the sell-off is attributed to a report that Goldman Sachs is dropping plans for a cryptocurrency trading desk. Another reason being given is the introduction of trading in bitcoin futures in early 2018, where we have seen massive market manipulation and suppression by experienced futures traders. There has also been an increase in crypto mining, the stealing of electricity and CPU cycles for blockchain mining, and crypto jacking which is essentially ransomware for bitcoins. A college student allegedly stole bitcoins by SIM swapping of mobile phones and stealing bitcoins through the mobile phone numbers of bitcoin investors. One of the crypto entrepreneurs that he hacked had just crowd funded $1million for an ICO. He was caught when his high spending life style drew attention and he was tracked through blockchain.

 

If terms like blockchain mining and ICO are new to you, you need to attend David Pinski's Blockchain session at NEXT 2018, September 25 to 27 in San Diego. David is the Chief Strategist for Financial Innovation and Head of the Hitachi Financial Innovation Laboratory. David’s breakout session is titled: Blockchain: Build Digital Trust in Your Enterprise. It is scheduled for Thursday 9/27 from 5:00 to 5:30.

 

David has a banking and technology background covering architecture, core banking systems, payments, strategy and venture with major banks such as Capital One, ING Direct and Bank of America. He has startup experience in areas of fraud, identity, loan origination with companies such as Zumigo, Silver Tail Systems, Octane Lending. And has been issued four patents in the area of fraud mitigation, payment networks and banking systems with a dozen more pending.

 

Hitachi's NEXT 2018 is the premier event for the digital revolution. It's for business leaders who embrace the digital economy and base decisions on data to ensure success. ... Come to NEXT 2018 to see what's next for your data center, data governance, insights and innovation, from Hitachi and our many partners. Register at NEXT 2018

Enterprise Storage Forum recently published their 2018 Storage Trends surveywhich made some interesting observations. The biggest challenges for IT and business leaders with operating their current storage infrastructure were aging equipment and lack of storage capacity.

Trendfocus.png

 

While much of the focus today is on high performance flash and NVMe, the critical concerns are still around aging gear, lack of capacity, high cost of operations, security, high maintenance costs, and silos before poor performance issues.

 

When one considers the data explosion being accelerated by Big Data, IoT, the increasing use of meta data, and AI/Machine learning, it is not surprising that storage capacity should be our greatest concern. You’ve heard analysts say that data is the new oil that fuels digital transformation. However, unlike oil that is consumed once, data will be consumed over and over again as long as we can store it. Machine learning gets more intelligent as it consumes more data, and that data can be reused for new learning models and analysis. In that sense data is more like gold in that it retains and may even increase in value. But unlike gold, data can be replicated many times over for protection, compliance, accessibility and many other requirements, driving the need for more and more storage.

 

According to a survey of organizations involved in Big Data, AI and IoT applications, conducted by NVMe analyst firm G2M, “Over 79% of respondents believe that current processing/storage architectures will not be able to handle the amount of data in their industry in the next 5 years.”

 

Trendfocus reports that 2Q 2018 HDD storage capacitygrew to 214.60 Exabytes, up 3%, breaking the previous record of 204.07 EB shipped in 1Q 2018. This growth was driven mainly by the nearline HDD market with sales reaching 14.15 million and totaling 104.35 EB in 2Q. Most of nearline capacity growth was driven by enterprise systems where 43% of nearline units had capacities of 10TB or higher. Total SSD capacity shipped in 2Q 2018 was an increase of 5% over 1Q and is estimated to be about 20 EB. SSD capacities are increasing with 3D NAND (capacity is increased by stacking the memory cells vertically in multiple layers), and the capacity price of an SSD is now almost equivalent to enterprise HDD at about $.25 to $.27 per GB.  Nearline HDDs are still much lower at $.02 and $.03 per GB which explains their growth in the storage market.  Most applications don’t require the higher performance of an SSD so a hybrid array with a mix of SSD, HDD, storage virtualization, and automated tiering to the cloud could be the best solution for capacity and aging infrastructure concerns.

 

Two of the key takeaways according to this 2018 storage trends survey referenced above are:

 

“Performance and cost drivers run neck-and-neck.
Balancing performance and cost aren't exactly new to storage administrators. What is new is the extreme storage performance that is now available with flash and flash accelerating technologies like NVMe. They are costlier than hybrid systems or lower cost AFAs, so it's important for IT to conduct detailed cost analyses for high performance storage purchases.

…..

Flash adoption is steady but not roaring ahead of HDD.
HDD's installed base is massive and works well with all but the fastest high-performance applications. It makes no sense for IT to rip and replace disk-based or hybrid systems - and may not for years to come. Flash/SSD is still the high-performance choice: important for transactional systems but not critical for most business applications.”

 

 

The net today is that capacity is more of a concern than performance and hybrid arrays with a mix of SSD for performance and HDD for capacity with cloud connectivity may be the best option for IT and business leaders who want to optimize their storage infrastructure.

 

For more information on what Hitachi Vantara has in store to to address your data and storage requirements, join us at NEXT 2018, September 25 to 27 in beautiful San Diego. The theme is" Your Data. Your Innovation."