It's increasingly recognized that DataOps will be an important framework for the next evolution of data management and will lead to profound organizational changes. In my recently published white paper, “How DataOps Transforms the Data Supply Chain,” I laid out how we at Hitachi Vantara think technology and the way we work will change as a result of DataOps.
First, the basics. DataOps, as envisioned by Hitachi Vantara, is organized into three interrelated disciplines:
- Agile Data Pipelines. DataOps is most analogous to DevOps here. In DataOps, data consumers must be able to find, assemble and transform data without expert intervention, creating the data pipelines to analytics, apps, systems and AI/ML models and move data into production.
- Data Governance. If everyone has access to all the data, DataOps would be a compliance and privacy nightmare. Controlled access with monitoring of who uses data and how it is used is crucial to internal governance processes and to complying with data-related regulations.
- Operations Agility. There must be hundreds of variations of data pipelines based on business need, with some batch ones that run once a week or once a month, while others are high-performance streams of real-time data delivered as fast as possible. DataOps will create a layer of technology and practices dedicated to operations agility that will break new ground in automation.
Each of these areas applied separately will bolster an organization. The white paper goes into more detail, but when applied together, these three disciplines will change the way we work with data in the following five ways.
1. Data Ownership
In most enterprises, data is typically owned and controlled by one department or function. Those who want to use data controlled by another group must request access. This is an example of the siloed thinking about data that DataOps seeks to eliminate.
In the world of DataOps, data is an asset that belongs to the entire company. There is a shared goal to find and use any relevant data that can help the business. While all can agree to this in principle, in practice it can be hard to implement. Many people will struggle to give up power and control over “their” data. Senior management must lead cultural change within the organization to demonstrate how the fluidity of data benefits everyone.
In DataOps, there must be security and controls on access, but the underlying desire must be to provide universal access to data. The default answer when someone requests data must be “yes” if it can be accomplished.
2. New Ways of Using Existing Skills
Many of the advanced practices that are part of the current environment will be used in DataOps, but in a different way. Data engineering, data quality, data profiling, data science and skills and processes for data management will all be vital and relevant in DataOps.
However, rather than being input to bespoke data pipelines, these skills are applied to create a DataOps infrastructure that assembles, describes, manages, transforms and delivers data to support a highly automated pull model.
3. Redefined Roles
DataOps will change how people operate. While it will not eliminate the roles people currently have in the data ecosystem, they must be able to adapt for DataOps to succeed.
Those who need data the most, namely the data and business analysts, will be more empowered to operate autonomously, from exploring data to preparing it. Yet data experts will also benefit from DataOps. There should be more collaboration between users and experts to improve models and data usage and management.
Data stewards will also be tasked with new responsibilities. They will need to ensure data quality and augment the richness of metadata. They will excavate data to be discovered, and their role may even evolve into a data reporter or researcher who proactively finds and suggests data for specific business needs.
Data engineers will still be vital but less overworked. They will focus on bringing data into this system and identifying gaps where data isn’t currently helping users, applying their expertise to solve specific problems. They will also be experts on the data itself and will be consulted in real-time as driven by the needs of the business. When automation and self-service reach their limits, data engineers and data scientists will be asked to help.
Finally, the IT and operations staff, as in DevOps, will be tasked to deliver a faster cycle of platform evolution and data supply chain deployment, instrumented by metadata and monitoring data. Foreseeably, OT based practices like reliability engineering could become part of IT and operations. Again, IT and operations will be less about running specific data pipelines (work that will move closer to the users) and more about maintaining and optimizing the DataOps infrastructure. The agility inherent in DataOps will permit IT and operations staff to rethink and refactor infrastructure, as well as other related processes.
4. Understanding and Managing Data as an Asset
As data becomes a core business asset, mindsets about it must change. It will also need to be managed differently. Business activity must be evaluated both from a financial perspective, as well as the value of the data acquired or created. This change in thinking will have profound effects on where money is invested.
In addition, companies must recognize the unique qualities of data as an asset. Data is not the new oil, an asset that is consumed when it is used. It is more like a new sun , a force capable of shining light and promoting growth everywhere.
5. Push Down Decision-Making
While the ideal of data flowing everywhere in the organization is promising, companies need to consider the implications. If data is flowing everywhere, it should make everyone smarter. But there must be strategies in place to address issues that will arise, such as whether users are required to get approval for every decision they make. DataOps will thus lead to a large cultural change that relates to decision-making. If data is flowing everywhere, it must be used to make decisions. #ThoughtLeadership#Blog
But this type of push down decision-making, where those using the data make the decisions, is a major cultural change for both those making the decisions and those overseeing them. Frequently, people at the edge of the business are not used to making decisions confidently because they lack the experience and often weren’t hired to do so. In addition, autonomous systems will play a role in pushing decision-making down and toward the edge. This type of change, where decision-making moves to the edge with more empowered people using data-based insights, alters the approach to management and oversight while also ultimately helping the company to grow decision-making “muscle” throughout the enterprise. Companies must be ready for this cultural change and have a plan for addressing it. In 2003, U.S. General Stan McChrystal began this type of thinking with his team of teams concept, in which the army ended siloed thinking and enabled the innovations of small teams to be spread throughout the entire organization.
While DataOps has a lot to do with technology, these changes in the way we work and enterprise culture may be the most profound impact of DataOps in the long term.