How to Create Valuable Data Tests | by Xiaoxu Gao | Jul, 2023

Information High quality dimensions

Taking a client viewpoint of knowledge high quality is undoubtedly a beneficial preliminary step. But it surely won’t cowl the completeness of the check scope. Intensive literature opinions have addressed this difficulty for us, providing a range of data quality dimensions which are related to most use circumstances. It’s advisable to assessment the record with knowledge customers and collectively decide which dimensions are relevant and create assessments accordingly.

| Accuracy     | Format           | Comparability     |
| Reliability  | Interpretability | Conciseness       |
| Timeliness   | Content material          | Freedom from bias |
| Relevance    | Effectivity       | Informativeness   |
| Completeness | Significance       | Stage of element   |
| Foreign money     | Sufficiency      | Quantitativeness  |
| Consistency  | Usableness       | Scope             |
| Flexibility  | Usefulness       | Understandability |
| Precision    | Readability          |                   |

You would possibly discover this record too lengthy and surprise tips on how to begin with it. Information merchandise or any data system could be noticed or analyzed from two views: exterior view and inside view.

Exterior view

The exterior view is about the usage of the info and its relation with the group. It’s usually thought-about a “black field” with performance to symbolize the real-world system. The size that fall into the exterior view are extremely business-driven. Typically, the analysis of these dimensions could be subjective, so it’s not at all times straightforward to create automated assessments for them. However let’s take a look at a number of well-known dimensions:

• Relevancy: The extent to which knowledge are relevant and useful for the evaluation. Contemplating a market marketing campaign geared toward selling a brand new product. All knowledge attributes ought to immediately contribute to the success of the marketing campaign comparable to buyer demographic knowledge and buy knowledge. Information like metropolis climate or inventory market costs are irrelevant knowledge on this case. One other instance is the extent of element (granularity). If the enterprise needs the market knowledge to be on the day degree, but it surely’s delivered on the weekly degree, then it’s not related and helpful.
• Illustration: The extent to which knowledge is interpretable for knowledge customers and the info format is constant and descriptive. The significance of the illustration layer is usually neglected when accessing knowledge high quality. It consists of the format of the info — being constant and user-friendly, and the which means of the info — being comprehensible. As an illustration, take into account a state of affairs the place knowledge is anticipated to be accessible in a CSV file with descriptive column descriptions, and the values are anticipated to be in EUR foreign money quite than in cents.
• Timeliness: The extent to which knowledge is recent for knowledge customers. For instance, the enterprise wants the gross sales transaction knowledge with a most delay of 1 hour from the purpose of sale. It signifies that the info pipeline must be refreshed incessantly.
• Accuracy: The extent to which knowledge is compliant with enterprise guidelines. Information metrics are sometimes related to sophisticated enterprise guidelines comparable to knowledge mapping, rounding modes, and many others. Automated assessments on knowledge logic are extremely beneficial and the extra, the higher.

Out of the 4 dimensions, on the subject of creating knowledge assessments, timeliness and accuracy are extra easy. Timeliness is achieved by evaluating the timestamp column with the present timestamp. Accuracy assessments are possible by way of buyer queries.

Inner view

In distinction, the inner view is worried with the operation that continues to be unbiased of particular necessities. They’re important whatever the use circumstances at hand. Dimensions within the inside view are extra technical-driven versus business-driven dimensions within the exterior view. It additionally implies that knowledge assessments are much less depending on customers and could be automated more often than not. Listed below are a number of key views:

• High quality of knowledge supply: The standard of the info supply considerably impacts the general high quality of the ultimate knowledge. The info contract is a good initiative to make sure supply knowledge high quality. As knowledge customers of the supply, we are able to make use of an analogous strategy to observe the supply knowledge as knowledge stakeholders do when evaluating the info merchandise.
• Completeness: The extent to which data is retained in its entirety. Because the complexity of the info pipeline will increase, there’s a greater chance of knowledge loss occurring inside the intermediate phases. Let’s take into account a monetary system that shops buyer transaction knowledge. The completeness check ensures that every one transactions efficiently traverse your complete lifecycle with out being omitted or overlooked. For instance, the ultimate account steadiness ought to precisely mirror the real-world state of affairs, capturing each transaction with none omissions.
• Uniqueness: This dimension goes hand-in-hand with the completeness check. Whereas completeness ensures that nothing is misplaced, uniqueness ensures that no duplication happens inside the knowledge.
• Consistency: The extent to which knowledge is constant throughout inside methods each day. The discrepancy is a standard knowledge difficulty that usually stems from knowledge silos or inconsistent metric calculation strategies. One other facet of the consistency difficulty happens between days when knowledge is anticipated to have a gentle development sample. Any deviation ought to elevate a flag for additional investigation.

It’s price noting that every dimension could be related to a number of knowledge assessments. What’s essential is knowing the suitable utility of dimensions to particular tables or metrics. Solely then, the extra assessments employed, the higher.

To date, we’ve mentioned the scale of exterior views and inside views. In future knowledge check designs, it’s essential to think about each the exterior and inside views. By asking the suitable inquiries to the suitable folks, we are able to improve effectivity and scale back miscommunication.