Our Chief Marketing Officer Jonathan Martin, published a blog to call attention to the fact that the old ways of data management are getting in the way of transforming data into Value. He called for customers to shift their strategies towards more collaborative, unified, and automated processes that better leverage data to deliver outcomes. The way to accomplish this is through DataOps, which he defines as data management for the AI era.
“It unleashes data’s ultimate potential by automating the processes that enable the right data to get to the right place at the right time – all while ensuring it remains secure and accessible only to authorized employees. We know it works because we use DataOps in our own operations.”
In the last half of last year we applied DataOps to consolidate our enterprise reporting systems into one data lake. 32 data domain owners from governance, pricing, services, procurement, partners, sales, marketing, supply planning and IT were involved. Collaboration was the key as they focused on these four pillars of our data management program.
Under the leadership of Ram Rao, Vice President, Technology and Analytics, and the collaboration of the domain users, the data lake was in production in 7 months and this data management program was in place. This was a major accomplishment due to the lack of documentation, legacy tribal knowledge, lack of awareness of the differences between operational and analytic reporting, multiple definitions from different domains, lack of basic data discipline, and whole new technology stacks for volume and scale processing. Key to the success of a data lake or DataOps program is the active participation of the business domain owners. You can’t sit back and let some one else make decisions for you about security, data definitions, and business rules and you can’t make these decisions in isolation. Once you define and agree on the business requirements, the data engineers, analysts and architects can help with DataOps tools to automate and deliver data quality, business intelligence and data architectures.
Business owners reported immediate benefits from this implementation. Fatima Hamad, Sr. Director, Strategic Pricing and Analytics, reported: “Leveraging our new Data Lake, we are enabling our Sales, Finance and Business users and customers to effectively track our performance and predict our forward-looking business health. Using the new, data-driven pricing guidance and approvals capabilities will allow more quotes to be approved at the account, district and country levels, while driving better pricing consistency, execution and outcomes for Hitachi Vantara”
We ran measurements across various use cases that included, Pricing Analytics, Master Data of Accounts, Demand Planning, and Data Integration Hub/ Data-as-a-Service (Supply Chain Services). The results speak for themselves:
- Improved data connectivity across 11 data sources from core to cloud
- The Data Catalog had over 10k data tables with 350k attributes
- Scalable processing with over 1000 ETL jobs a day
- Platform scalability enabled 1.8M queries over 6 months
- Enhanced analytics for 110+ users with 2.2k executions over 6 months
DataOps is the Art of Harmonizing People, Process and Technology
DataOps is a process-oriented methodology that combines existing tools like data warehouses and data lakes with new technologies like AI and ML. And, like DevOps, it merges analytic, data, and business teams together to improve quality and predictability of business decisions and reduce time to value. One of the key enablers for the DataOps process was Pentaho’s data integration and pipeline orchestration. However, the primary key to success is leadership and the active participation of the business domain users, working with the data engineers, data scientists, and data analysts.
Greater harmony between your people, processes and technology means greater value from your data. Visit our website to see how you can gain your DataOps advantage today.