AutoZone

Today, there is no boardroom meeting or a simple business conversation where the word “Data” is said at least a few times - most likely, a most used word. From decisions to integrating processes, reliance on data is growing by the day. It is not wrong to say that “Data is now ubiquitous” in modern business with ever-expanding need to process data faster and deliver more actionable insights in near real time.

The value of data lies in making better decisions, improving the efficiency of operations, or generating new revenue streams impacting a business’s P&L. Data-driven decision-making is dependent on collecting data that leads to extracting patterns, facts, or insights. But what if the data is inaccurate, or historical data is not a good indicator of the future? Bad data will result in erroneous decisions leading to negative business outcomes.

Let's take an example to showcase the importance of data quality using a retail customer scenario. Most reward programs offered by retailers depend on the customer providing information to sign up for the reward program. What if the sign-up process does not do the due diligence in identifying if the customer is already enrolled in the reward program? This event will result in multiple reward program identifiers assigned to the same customer creating a lot of friction not only in how we interact with the customer in real-time but also making erroneous decisions.

The consequences are not limited to higher costs attributed to managing duplicate identifiers but may also lower the effectiveness of promotions offered and decisions made for customer retention.

To drive value from data, the data users/data scientists will enrich and wrangle the data themselves within the frame and needs of their use case. While the data scientist responsible for making purchase recommendations to loyalty customers may deduplicate/unify customer records, the data scientist responsible for rewarding sales may not worry about deduplicating customers. These result in data silos without data producers realizing the impact of data quality issues.

Hence, data quality is not only the responsibility of the data producers but should be the responsibility of everyone who touches data through the entire value lifecycle. Ideally, the data quality needs to be considered as a “recurring audit,” “continuous data quality,” and then eventually providing “feedback to data producers.”

Recurring Audit (RA) is the process where data users audit the data for quality at an agreed-upon cadence. There are many tools and platforms available in the marketplace that provide Artificial Intelligence / Machine Learning based data capabilities to unify and correct data. RA should also result in the identification of quality rules that are provided as feedback to data producers to correct the problem from happening again and to the data engineers to build data correction within the data pipelines that feed the data ecosystem.

Continuous data quality (CDQ), on the other hand, is the process that monitors data for anomalies, schema, and data drift as it is ingested in the data ecosystem. CDQ will filter or flag the data for possible issues to be investigated by data users. During their investigation of the issues, data users either accept the anomalies and drift or identify additional quality rules for RA.

If adopted across the organization, the two processes should reduce data silos and erroneous decision-making. In addition, data producers should prioritize implementing the quality rules at the entry point or data production.

To derive the full value of data, the quality of data becomes the most essential. Without ensuring that the data is complete, relevant, timely, accurate and consistent, data users will spend large amounts of time cleaning the data in their own functional silos, wasting time and effort, and creating confusion. It is the quality of data that is important, not the quantity.