In my previous blog post, “DataOps: Optimizing the Data Experience,” I explored the concept of DataOps as the catalyst to produce successful outcomes while undergoing digital transformation initiatives. The DataOps functions and capabilities dramatically shift the way in which data can be managed and analyzed by changing the creation of dataflows from an engineering activity to an operational activity.
In this shift, IT experts can focus their efforts on tools and automation that enable functional data users to become more autonomous and dramatically improve speed and overall performance in the way they work.
For example, let’s look at the work required to install RFID tags in the pipelines to transport gasoline from refineries to large storage terminals and identify variations of pressure and the possible causes. As is the case in most enterprises, data scientists would have to wait several months for the following actions to be completed:
- Data engineers need to understand and define the initial concept for a data model to support this process.
- Data engineers collect and map the data, potentially in an automated process.
- Data architects design how the new data fields fit into the canonical enterprise data model and master customer data model.
- Data stewards identify and define the items in the business glossary.
- Data operations produce a catalog with the new metadata information.
- Data governance decides what standards to apply to the new data approval and establishes permissions and access control.
- Data security operations need to unlock the new data and make it available, traceable, and secure.
In some instances, especially in large corporations, it might be typical for the data scientist to have access to the data within six months. If the data scientists were able to reduce that time by performing some of the data integration themselves and potentially bypass some of the normal process, they may be able to cut the time down. However, I’m not suggesting that anyone should skip any processes established by their organization.
Instead, what if the DataOps team could step through all the processes (without bypassing the architecture, design, governance, or security functions and capabilities), and deliver the data within a few days? Or, what if they could deliver it the same day that RFID was first implemented as real-time streaming data? Would that create a fundamental change?
In today’s operating reality, we’re faced with having to wait months for IT to architect, design, test, and implement integrations for new systems or new versions of applications. This is especially the case if those applications reside in different environments (on-prem, co-location facilities, and in the public cloud).
To illustrate how long this historically takes, let’s look at the typical time needed to integrate two companies. For instance, one of my former employers acquired companies to penetrate different market adjacencies. It entailed managing the integration between the core business of the company and the new capabilities and applications that supported them and were added as a result of the acquisition. It took several months to rationalize and integrate the data from the new companies and compare it to the existing data model.
Imagine if DataOps could facilitate it in one-tenth of the time. The cost savings and financial and competitive advantage for mergers like these could be worth millions of dollars. My experience over the years in process automation suggests that DataOps practices can truly achieve this breakthrough.
For another example of common data experiences, let’s look at typical software bugs or data errors. These happen daily in the typical way that data variation appears such as due to technology changes, regulatory changes, system changes, and even simple data errors. A frequent example is a retail business making pricing errors. Consider these examples:
- A British multinational retailer offered 50-inch 3D plasma screen TVs for £199 rather than £1,099.
- A British multichannel retailer of trade tools, accessories, and hardware mistakenly priced a ride-on mower for £34.99 instead of £1,599.99.
- A supermarket chain offered a $50 voucher and forgot the “first time only” statement; hundreds of shoppers used it repeatedly until the company ended the promotion.
- A wholesale retailer slashed prices on everything from mountain bikes to high-definition TVs and eventually canceled hundreds of sales and offered shoppers a $10 gift card instead.
In these and many other cases, companies recovered from the data issues and managed to contain their costs, but they still incurred a net financial loss and had long-term customer experience issues.
With DataOps in place and the right monitoring and alerting, these kinds of issues could be resolved in minutes. And without any impact on customers. With DataOps, you can eliminate these kinds of errors.
In subsequent blog posts, we’ll examine how to implement a DataOps approach in the organization. We’ll also look at technologies you can implement to support these practices and make an impact.
