DataOps With VSP Smart Storage

By Hubert Yoshida posted 03-06-2020 01:58


DataOps is data management for the AI era. It unleashes data’s ultimate potential by automating the processes that enable the right data to get to the right place at the right time—all while ensuring it remains secure and accessible only to authorized employees. DataOps is not a product, service or solution. It's a methodology: a technological and cultural change to improve your organization's use of data through better collaboration and automation. DataOps is a wide suite of enabling technologies and processes including data management technology (data catalogs, data virtualization, data pipelines, AI model management) as well as technologies for versioning, test automation, deployment automation and release management, and runtime orchestration, collaboration, self-service and governance. That means improved data trust and protection, shorter cycle time for your insights delivery, and more cost-effective. Organizations understand the need to use next-generation technology, like containers for increased agility and artificial intelligence to increase insight into growing data sets, but legacy infrastructure and workflow complexities are preventing them from recognizing the value they hope to gain from these strategic investments.


VSP Smart Storage With AI and ML

The good news is that infrastructure, like everything else has gotten smarter. AI and ML are being incorporated into our VSP storage systems, to offload a lot of the complex processes to select, deploy, and/or provision resources; the length of time needed to deploy proper IT services; the inability to provide adequate security and compliance controls; and the lack of skills to support emerging technology such as AI and data science. This offloads a lot of the grunt work and skill sets which prevents IT from investing resources and talent into a modern DataOps platform.  This starts with the architecture of our VSP storage controller.



Having a smart storage systems is not very useful if you cannot tap into the “smarts”.  In order to leverage this smart capability and integrate it with DataOps data management, a REST API is built directly into our VSP storage controllers. REST stands for “representational state transfer”. APIs built according to REST architectural standards are stateless, which means neither the client nor the server need to remember any previous state to satisfy it. Stateless components can be freely redeployed if something fails, and they can scale to accommodate load changes. REST enables plain-text exchanges of data assets. It also makes it possible to inherit security policies from an underlying transport mechanism. REST APIs provide a simplified approach to deliver better performance and faster paths to develop, deploy, and organize. We increased the memory and CPU in the controller specifically to support the REST API running natively in the controller. This gives us the opportunity to not only connect with other vendor’s management stacks, but also apply analytics, machine learning and automate deployment of resources through REST APIs. Here are some examples of how this API strategy brings operational benefits to the Smart Data Center. And frees up resources for DataOps.


Infrastructure Analytics

Hitachi Vantara has developed an IT analytics tool, Hitachi Ops Center Analyzer that can provide prescriptive and predictive analytics by mining telemetry data from servers, storage appliances, networking systems and virtual machines to optimize performance, troubleshoot issues and forecast when a business may need to buy new storage systems or capacity. There are 77 performance metrics that we can provide via REST API over IP connections. Based on an analysis of these metrics the Ops Center Analyzer can determine the right actions to take, then launch into an automation tool to invoke the appropriate services to execute that action.




The automation tool, Hitachi Ops Center Automator, contains a catalog of templates that can automate and orchestrate the delivery and management of IT resources. The analytics tool can communicate with the automation tool, through a REST API, to help you select the right template, fill in the parameters and request deployment of resources. During the execution, the automation tool may need to communicate with third party switches, virtual machines, containers or public cloud through their APIs. When one considers all the tedious steps required to request and deploy storage, networking, hypervisor, and application services for hundreds or even thousands of users, you can see how automation can reduce days of work down to minutes.



Hitachi Ops Center Automator has a catalog of canned, application provisioning templates which we are continuing to expand. This internal “App” store of provisioning packages includes Hitachi Storage Orchestration, Provisioning, Flash Module Compression (FMC) Optimization, Creation of a Virtual Storage Machine (VSM) that spans two physical storage systems for active/active availability, Replication (2DC, 3DC, GAD), SAN Zoning: Brocade BNA, Cisco DCNM, Oracle DB Expansion, VMWare Datastore Life Cycle Management, and Plugins/Utilities: CMREST (Configuration Management Rest API), JavaScript, OS, VM, OpenStack, AWS, etc.


Policy based Copy Management

Since most data is backed up and copied, a copy data management platform is available to simplify creating copies and managing policy-based workflows that support business functions with controlled copies of data. Hitachi Vantara provides Hitachi Data Instance Director, which is the data protection automation tool you can invoke to deploy the copy workload and set up and enforce data protection SLA policies through REST APIs.


IT Service Management

Hitachi Ops Center’s REST API is open and available for working with third party resources. Enhancements to the software include integration with IT service management (ITSM) tools, including the ServiceNow platform, for better IT resource tracking and improved REST API integration for working with third-party resources. Hitachi Ops Center Automator creates workflows to use with ServiceNow, approval can be administrator driven or driven by the Ops Center Automator.  Hitachi Ops Center Automator executes the automated workflow changes and updates the ticket.




Third Party and Home Grown Services

Hitachi Ops Center Automator encourages working with third party services by providing a design studio, and a developer community site.  Hitachi Ops Center Automator service builder is the design studio where users have the flexibility to create their own or customize a service template to fit their own environment, operation policy and workflow. They provide the capability to leverage 3rdparty or home grown tools.




Hitachi Vantara has launched the Hitachi Ops Center Automator Community Site.

It is available to external users. Here Hitachi Vantara shares sample service templates (more than 30 content packs), prototypes, how to use Hitachi Ops Center Automator (formerly named Hitachi Automator Director), Q&A, etc. Here we will be collaborating with customers / partners to develop more workflow automation and orchestration content.


Call Home Monitoring

Other uses of REST APIs include our call-home monitoring system, Hitachi Remote Ops. which has been re-coded to use our native REST APIs to collect information about the storage systems and report that back to our support teams. Hitachi Remote Ops provides 24/7 monitoring for early alerting and insight to help your storage system run at peak efficiency. Only authorized Hitachi Vantara support specialists may establish a connection with your site, and only by using the Hitachi Vantara internal network. Secure access with encryption and authentication keeps error and configuration information tightly controlled, and your production data can never be accessed.


Container Plug-In

We have a Hitachi Storage Plug-in for Containers that integrates us with Docker and thereby with Kubernetes and Docker Swarm. This Plug-in is built on the REST API that is also available to customers to integrate with. This plug- in retains the state of the storage as containers are spun up and down. Without this, the storage for a container would disappear when the container goes away.


VSP Configurator

The VSP storage configuration tool, Hitachi Ops Center Administrator, can be accessed through software on an external virtual or physical server via the REST API.


Virtualization of Legacy Storage

Legacy storage infrastructure that was never intended to support the demands of modern IT, need not be a hinderance. Since the REST API is built into the VSP controller, legacy storage systems that are virtualized behind the VSP controller inherit all the benefits of the VSP. Keep your legacy storage and use it as part of your automated pool of tiered storage.




The use of REST APIs are key to the integration of infrastructure, software, and analytics to create an intelligent data center that enables IT to focus on DataOps and the modernization of the data center. A Rest API built directly in our VSP controller provides connection with other vendor’s management stacks and enables the application of analytics and machine learning for automated deployment of resources. Benefits include:


Reduce workloads from days to minutes

Reduce errors resulting from tedious manual work

Reduce the need for skilled IT staff

Optimize use of IT resources

Increase speed of resolution to customer requests

Effective Outage Management with quicker return to service

Customize to fit an organization’s specific environment

Improve forecasting of future resource requirements


 Hitachi’s management strategy is based around the idea of a shared and open API architecture that allows us to simplify transmission of data across our suite of management tools & 3rdparty tools. Everything we have is API-based so that we can draw information in from other sources to create a more intelligent solution, but we can also pass information out if we’re not the master in the environment, so we can make other things smarter. In addition, our goal is to enrich our ‘library’ of 3rdparty device API information so that we can capture and analyze telemetry data from a broad range of devices & interact with them. We are taking a very vendor-neutral approach as we recognize that there is a much broader opportunity to deliver better solutions if we integrate with more vendors and partners. Our DataOps advantage is based on an open strategy