Why DataOps Will Create a New Data Culture

By Lothar Schubert posted 08-12-2020 04:11


Culture is created by all of the actions we take over and over again that serve to reinforce ideas, habits, and practices. At some point we share those ideas and practices throughout an organization or community and call it culture. Then, when new challenges present themselves, we face them using those ideas, habits, and practices.

At Hitachi Vantara, we see the arrival of DataOps as the beginning of a new cycle of culture creation that will permeate all areas of an enterprise.

A recent whitepaper called “Is DataOps Your Windfall of Value?” describes the technology of DataOps and how DataOps, as envisioned by Hitachi Vantara, is organized into three interrelated data disciplines:

  • Agile Data Pipelines. In DataOps, data consumers must be able to find, assemble and transform data on their own, without the help of data experts, creating data pipelines to feed everything from analytics to AI/ML models.
  • Governance and Measurement. For DataOps to work in a business context, data access must be controlled strategically with internal governance processes to comply with data-related regulations.  The flow of data through DataOps pipelines must be measured so the data supply chain can be monitored and optimized.
  • Operations Agility. DataOps infrastructure must be able to handle a variety of data pipelines with a new layer of technology and practices dedicated to operations agility that will break new ground in automation.
At first companies will take up these technologies and have in mind a general sense of what DataOps means. But as technology changes the way people work, new ideas, habits, and practices will take hold. In other words, a new culture will be born. The whitepaper “The Cultural Impact of DataOps: Collaboration, Automation and the War on Silos” offers a speculative look at the following seven dimensions of DataOps culture:
  • Collaboration
  • Automation and a metadata mindset
  • Data as a shared asset
  • End-to-end design thinking
  • Enlightened and guided empowerment
  • Silo paranoia
  • Push down decision-making

Here is a brief tour of what a DataOps culture may look like.

Collaboration in the world of DevOps is simpler than it will be in DataOps. DevOps joins two disciplines, but the goal is that everyone across the organization uses data. DataOps encompasses everyone from the beginning of the data supply chain where data originates (from IoT devices to enterprise applications to massive third-party repositories such as the Open Data Initiative) to all the people who model and blend data all the way to those who put it to use in applications and analytics. Ultimately, users must be able to identify problems and collaborate to pull applicable data together while working with data experts as needed. This model of allowing people to use policy-based systems versus one-off requests shouldn’t hinder collaboration; instead it should foster collaboration on more meaningful problems and reduce grunt work across the board.

Automation and Metadata Mindset
The world of DataOps embraces automation, with metadata as its foundation to facilitate broad use of the latest and most accurate data, eliminating the frustration and delays that led people to use spreadsheets. 

With DataOps, users are going to want the same type of operational simplicity and rapid iteration that occurred in DevOps, but will need greater automation to achieve it. That automation will largely be driven by metadata descriptions, including descriptions of data, processes used to transform data, and metadata captured from monitoring of the use of data. This greater use of automation will lead to greater reliance on autonomous AI and machine-learning systems that use metadata as their fuel. 

Understanding Data as a Shared Asset
Everyone talks about generating and using data, but its economic value is less frequently explored (see, for example, Hitachi Vantara’s recent joint research project conducted with the University of San Francisco). Unlike other assets, data does not depreciate, becoming less valuable the more it is used. Instead, data gains value over time, increasing in applicability and accuracy as it is used and refined. Data is not the new oil; it’s far more like sunlight. Its power can radiate and shed light on different aspects of a business all day long, all week long, all year long. 

DataOps will break down the organizational boundaries that fostered exclusive ownership by departments or individuals around certain types of data that resulted in a proliferation of data silos. But the instinct to control data in this way won’t disappear overnight, as data is a valuable resource and a source of power. Companies will need to govern, control, and secure data sharing and access, while also preventing data hoarding so that new data silos don’t develop. Otherwise, DataOps won’t work.

End-to-End Design Thinking
One way to understand both DevOps and DataOps is as an application of design thinking. In both DevOps and DataOps the whole problem, from end-to-end, including all the goals, is rethought. DataOps makes organizations think through the flow of data from its creation to its use, which reaches the entire business, as every part relies on data. Companies have to adapt from a world of only specialities to one in which data serves larger automated systems that allow people to create and maintain data supply chains to operate efficiently. 

Enlightened and Guided Empowerment
The chief objective of DataOps is to involve more people in data and allow them to do more work supported by automation and intelligent systems. That is a form of self-service, but the definition of that term must be broader than in the past to achieve guided and enlightened empowerment. Users must have a simpler way to get the data they need, with inherent security and governance. Technology must become less complex while automation increases, as users get suggestions and guidance. Self-service should also increase, but not impede collaboration with experts. Data experts must provide help when it is needed. Enlightened and guided empowerment must become a cultural value for DataOps to succeed.

Silo Paranoia
Both DataOps and DevOps seek to break down silos that emerged naturally between parts of the organization. The impulses that created these silos will not disappear overnight, whether it’s attempting to control assets or gain power. Part of DataOps culture is staying vigilant and on the lookout for silos that may begin to reemerge. 

Push Down Decision-Making
The impact of a greater spread of data in an organization will have its own cultural impact. If data is flowing everywhere and informing more people, it is logical to assume that they will be empowered to make decisions based on that data. But this is a marked shift for both managers and the end users making the decisions in many cases. Organizations must be prepared for this cultural evolution and have a plan to handle it for DataOps to succeed. 

DataOps reduces the friction around data. As data flows more freely through an organization, informing decisions and answering questions, the dynamics of the culture of that business will change as DataOps breaks down organizational boundaries around data.

As indicated above, however, that doesn’t mean that the creation of a data culture will be entirely automatic or conflict-free. But as data flows more freely, and questions can be answered quickly, the inherent value of data will spark positive changes in areas that cultivate the cultural changes facilitated by DataOps. Clear success in one business unit, brought about by DataOps, will foster change in other areas. To learn more about DataOps, read “Is DataOps Your Windfall of Value?

1 comment



05-05-2022 13:26

Nice Article