Ask ten people in your organisation if your data is good enough, and you’ll get ten different answers. Ask them how they would measure it, and the room might grow very quiet.
Awareness usually isn’t the challenge with Data Quality, but the hard part is precision: knowing what to measure, how to set meaningful targets, and who should act when quality drops. Part of this difficulty is a terminology problem. Terms like dimensions, metrics and KPIs are used interchangeably in conversations, while they mean different things and serve different purposes.
In this blog post, we’ll explain the vocabulary and give you a practical guide on the foundational concepts and development of KPIs that result in real change in your organisation.
The language of Data Quality
Before we can measure Data Quality, we need to develop a shared vocabulary on some important Data Quality concepts. Terms like dimension, metric and KPI are often confused or used interchangeably while they have different meanings.
Data Quality dimension:
A Data Quality dimension is a high-level category of quality that helps group your more specific measures. Dimensions give you a lens through which you can assess the quality of your data, and a shared language to help structure conversations between business, data and technology teams. In another blogpost we discuss the most used Data Quality dimensions, so we’ll only give a quick overview here.
- Accuracy: Measures if your data is correct compared to real-world values.
- Completeness: Measures the presence of required data, assesses if data is missing or omitted.
- Consistency: Checks for conflicts between different data elements, either within or between datasets.
- Timeliness: Ensures data is relevant, updated regularly and available when required.
- Validity: Data should adhere to rules and standards, conforming to formats, values and ranges.
- Uniqueness: Checks if a dataset has duplicate records containing redundant information.
If you want to read more about Data Quality dimensions, check out our blog post on this topic.
These dimensions are intentionally broad. They’re designed to be applied across industries and use cases. Dimensions are usually not measured directly, but are rather the categories that your metrics sit underneath. A single dimension might have dozens of specific measurements depending on your datasets and use case.
Data Quality metric:
Metrics on the other hand are very concrete calculations. They turn dimensions into something measurable on a specific dataset, and have three main components.
A dimension the metric belongs to. For example, a metric can belong to the accuracy dimension.
A dataset or data element the metric applies to. This could be your CRM or financial dataset.
A calculation that produces a number. The calculation itself can take many forms, like percentages, counts or ratios. An example of an accuracy metric could be the percentage of customer records where the email field is not null or empty.
These metrics are inherently technical and are usually written as queries or rules in your Data Quality tool of choice. They are also very context-specific, as two organisations measuring validity will likely define completely different metrics, because they have different datasets, systems and definitions of what validity means for their use case.
But metrics are still just numbers. Without business context, an 82% uniqueness score does not tell you whether to celebrate or escalate. For that, we need KPIs.
Data Quality KPI:
A Data Quality Key Performance Indicator (KPI) is a metric that has been enriched with business context. Every KPI is a metric, but not every metric has to be a KPI. Only the ones that are directly tied to business outcomes that are worth tracking and acting on, making them management instruments instead of just a measurement.
Strong Data Quality KPIs have five main elements:
- Metric: What is being measured and how.
- Threshold: What level of quality is required, could be > 95%.
- Owner: Who is responsible for maintaining quality and acting when it drops below the set threshold.
- Review cadence: How often the KPI is evaluated and by whom.
- Consequence: What happens when the threshold is breached, could be an alert, review or escalation.
Organisations often have plenty of metrics sitting in Data Quality dashboards, numbers that are monitored but never acted on. A metric becomes a KPI when it is embedded in a governance process: the standard is defined, someone is accountable, and there is a clear escalation process.
Looking back at our accuracy metric example, this can become a KPI if we define that the percentage should be above 95%, the customer information dataset owner is responsible, the KPI is reviewed weekly and there is a clear escalation process that notifies the technical team of a potential issue.
This shift from measurement to accountability is what separates a data quality monitoring exercise from a complete data quality management programme. So how do you define these KPIs for your organisation?
How to set up good Data Quality KPIs
Start with business outcomes, not data
One of the most common mistakes in Data Quality is reasoning from “what can we measure in the data?” instead of “what business decisions depend on the data being of high quality?”. Your Data Quality engineers can come up with countless metrics, but if those are on fields that are not used in reporting, they will not add any value. So instead, ask yourself what business processes and decisions depend on data being correct, valid, consistent, and start there.
Select the right dimensions for the domain
Not all Data Quality dimensions are created equal. According to the widely used DAMA DMBOK standard, there are 60 different dimensions that you can measure Data Quality on. In practice however, most organisations only use 4-6 dimensions. And not all dimensions matter equally in every context. In a financial reporting dataset, accuracy and timeliness are critical, while in a marketing list completeness and uniqueness matter more. Selecting the right dimensions for your use case is critical in narrowing down the scope of your metrics. As a starting point, review your critical business processes and ask which quality failures would cause the most damage.
Define thresholds that mean something
KPIs without thresholds are not really KPIs. When you add a threshold to your metric, you define when the data is good or bad, and when you should take action. Thresholds can be based on business impact, historical baselines in your data or industry benchmarks. Remember to document how you came to a certain threshold, so you can revisit it later if your KPIs do not work as expected. This documentation also increases trust in the KPI, and transforms it from ‘just a number someone picked’ to a business decision.
Assign ownership
Data Quality KPIs fail to bring value to your organisation if nobody is accountable for them. By assigning someone like a business process owner or data steward in your organisation as responsible for monitoring the KPI, and someone for taking action when it’s breached, you create a network of trust and accountability behind your KPIs. The consequences of breaching a threshold should also be clear: do you file a high-priority ticket with a technical team, or does a data cleaning workflow start automatically. This connects your governance policies to the operational reality of your organisation.
Build for review, not just monitoring
KPIs don’t add any value when nobody looks at the dashboard they are on. It is important to embed them into your governance rhythm: when are they reviewed, by whom, what triggers escalation, when should revisions be made? Putting your KPIs in dashboards helps their visibility, but it’s the governance process around them that makes them work. Regularly reviewing your KPIs in data council or domain review meetings is a great starting point.
Conclusion
This framework is simple but effective once it falls into place. Dimensions give you the vocabulary, metrics provide the measurement and KPIs assign accountability. Most organisations have the first. Many have the second. But far fewer have all three working together in a meaningful way that drives improvement. That is exactly the gap a Data Quality Management programme is supposed to close. Getting there does not require the perfect tool or a mature governance programme to get started. Beginning with a small set of well-designed KPIs, is an effective way of building momentum and growing the case for investing further.
Do you have Data Quality metrics, but they aren’t driving change in your organisation? Have a look at our Data Quality services or get in touch to see how we can assist your business!
