Why Probabilistic Linkage is More Accurate than Fuzzy Matching or Term Frequency based approaches | by Robin Linacre | Oct, 2023

How successfully do completely different approaches to report linkage use data within the data to make predictions?

A pervasive knowledge high quality drawback is to have a number of completely different data that discuss with the identical entity however no distinctive identifier that ties these entities collectively.

Within the absence of a singular identifier corresponding to a Social Safety quantity, we will use a mixture of individually non-unique variables corresponding to identify, gender and date of beginning to establish people.

To get the most effective accuracy in report linkage, we want a mannequin that wrings as a lot data from this enter knowledge as doable.

This text describes the three sorts of data which can be most essential in making an correct prediction, and the way all three are leveraged by the Fellegi-Sunter mannequin as utilized in Splink.

It additionally describes how some various report linkage approaches throw away a few of this data, leaving accuracy on the desk.

The three sorts of data

Broadly, there are three classes of knowledge which can be related when making an attempt to foretell whether or not a pair of data match:

1. Similarity of the pair of data
2. Frequency of values within the total dataset, and extra broadly measuring how frequent completely different eventualities are
3. Information high quality of the general dataset

Let’s have a look at every in flip.

1. Similarity of the pairwise report comparability: Fuzzy matching

The obvious option to predict whether or not two data symbolize the identical entity is to measure whether or not the columns include the identical or comparable data.

The similarity of every column will be measured quantitatively utilizing fuzzy matching capabilities like Levenshtein or Jaro-Winker for textual content, or numeric variations corresponding to absolute or proportion distinction.

For instance, Hammond vs Hamond has a Jaro-Winkler similarity of 0.97 (1.0 is an ideal rating). It is in all probability a typo.

These measures might be assigned weights, and summed collectively to compute a complete similarity rating.

The method is usually often called fuzzy matching, and it is a crucial a part of an correct linkage mannequin.

Nonetheless utilizing this method alone has main disadvantage: the weights are arbitrary:

• The significance of various fields needs to be guessed at by the consumer. For instance, what weight needs to be assigned to a match on age? How does this examine to a match on first identify? How ought to we determine on the dimensions of punitive weights when data doesn’t matches?
• The connection between the power of prediction and every fuzzy matching metric needs to be guessed by the consumer, versus being estimated. For instance, how a lot ought to our prediction change if the primary identify is a Jaro-Winkler 0.9 fuzzy match versus a precise match? Ought to it change by the identical quantity if the Jaro-Winkler rating reduces to 0.8?

2. Frequency of values within the total dataset, or extra broadly measuring how frequent completely different eventualities are

We are able to enhance on fuzzy matching by accounting for the frequency of values within the total dataset (typically often called ‘time period frequencies’).

For instance, John vs John, and Joss vs Joss are each precise matches so have the identical similarity rating, however the later is stronger proof of a match than the previous, as a result of Joss is an uncommon identify.

The relative time period frequencies of John v Joss present a data-driven estimate of the relative significance of those completely different names, which can be utilized to tell the weights.

This idea will be prolonged to embody comparable data that aren’t a precise match. Weights can derived from an estimate of how frequent it’s to watch fuzzy matches throughout the dataset. For instance, if it’s actually frequent to see fuzzy matches on first identify at a Jaro-Winkler rating of 0.7, even amongst non-matching data, then if we observe such a match, it doesn’t supply a lot proof in favour of a match. In probabilistic linkage, this data is captured in parameters often called the u chances, which is described in additional element here.

3. Information high quality of the general dataset: measuring the significance of non-matching data

We’ve seen that fuzzy matching and time period frequency based mostly approaches can enable us to attain the similarity between data, and even, to some extent, weight the significance of matches on completely different columns.

Nonetheless, none of those methods assist quantify the relative significance of non-matches to the anticipated match chance.

Probabilistic strategies explicitly estimate the relative significance of those eventualities by estimating knowledge high quality. In probabilistic linkage, this data is captured within the m chances, that are outlined extra exactly here.

For instance, if the information high quality within the gender variable is extraordinarily excessive, then a non-match on gender can be sturdy proof in opposition to the 2 data being a real match.

Conversely, if data have been noticed over a variety of years, a non-match on age wouldn’t be sturdy proof of the 2 data being a match.

Probabilistic linkage

A lot of the ability of probabilistic fashions comes from combining all three sources of knowledge in a means which isn’t doable in different fashions.

Not solely is all of this data be integrated within the prediction, the partial match weights within the Fellegi-Sunter mannequin allow the relative significance of the several types of data to be estimated from the information itself, and therefore weighted collectively accurately to optimise accuracy.

Conversely, fuzzy matching methods usually use arbitrary weights, and can’t totally incorporate data from all three sources. Time period frequency approaches lack the power to make use of details about knowledge high quality to negatively weight non-matching data, or a mechanism to appropriately weight fuzzy matches.

The creator is the developer of Splink, a free and open supply Python bundle for probabilistic linkage at scale.