Building Durable Data Pipelines. Data engineering techniques for robust… | by 💡Mike Shakhomirov | Mar, 2024

Knowledge engineering strategies for strong and sustainable ETL

Knowledge sturdiness in knowledge pipeline design is a widely known ache level within the knowledge engineering house. It’s a well-known indisputable fact that knowledge availability and knowledge high quality points can result in a major enhance in time on non-value-added duties. On this story, I want to discuss knowledge engineering design patterns for knowledge pipelines to make sure knowledge is at all times there. We’ll discuss strategies that may assist us to construct a sustainable knowledge transformation course of the place knowledge is at all times delivered on time and our knowledge pipeline may be described as strong, sturdy and possibly even self-fixing.

If an information pipeline fails staff most certainly should carry out a set of handbook duties together with pointless knowledge sourcing, aggregation and processing to get to the specified end result.

Knowledge sturdiness is a famend danger consider knowledge engineering. In my view, it’s the least mentioned matter on-line in the intervening time. Nonetheless, merely since you don’t see the issue it doesn’t imply it isn’t there. Knowledge engineers won’t converse of it typically. The problem although exists, seeding worry amongst knowledge practitioners and turning knowledge pipeline design into an actual problem.

Knowledge availability and knowledge high quality points may result in additional delays in knowledge supply and different reporting failures. Based on McKinsey report, time spent by staff on non-value-adding duties can enhance drastically as a result of these components:

This might usually embody not-required knowledge investigations together with additional knowledge sourcing, knowledge cleaning, reconciliation, and aggreagtion leading to a lot of handbook duties.

These handbook duties are completely pointless

So how can we construct strong, sturdy and self-fixing pipelines?

What’s an information pipeline?

There’s a knowledge pipeline each time there may be knowledge processing between factors A and B. As soon as may be thought of because the supply and the opposite as a vacation spot: