Traditional vs. Neural Metrics for Machine Translation Evaluation

100+ new metrics since 2010

An analysis with automated metrics has the benefits to be sooner, extra reproducible, and cheaper than an analysis performed by people.

That is very true for the analysis of machine translation. For a human analysis, we might ideally want professional translators

For a lot of language pairs, such consultants are extraordinarily uncommon and tough to rent.

A big-scale and quick handbook analysis, as required by the very dynamic analysis space of machine translation to guage new techniques, is usually impractical.

Consequently, automated analysis for machine translation has been a very lively, and productive, analysis space for greater than 20 years.

Whereas BLEU stays by far essentially the most used analysis metric, there are numerous higher alternate options.

Since 2010, 100+ automated metrics have been proposed to enhance machine translation analysis.

On this article, I current the preferred metrics which might be used as alternate options, or as well as, to BLEU. I grouped them into two classes: conventional or neural metrics, every class having completely different benefits.

Most automated metrics for machine translation solely require:

• The translation speculation generated by the machine translation system to guage
• Not less than one reference translation produced by people
• (Hardly ever) the supply textual content translated by the machine translation system

Right here is an instance of a French-to-English translation:

Le chat dort dans la delicacies donc tu devrais cuisiner ailleurs.

• Translation speculation (generated by machine translation):

The cat sleeps within the kitchen so cook dinner some other place.

The cat is sleeping within the kitchen, so you must cook dinner some other place.

The interpretation speculation and the reference translation are each translations of the identical supply textual content.

The target of the automated metric is to yield a rating that may be interpreted as a distance between the interpretation speculation and the reference translation. The smaller the gap is and the nearer the system is to generate a translation of human high quality.

Absolutely the rating returned by a metric is often not interpretable alone. It’s virtually all the time used to rank machine translation techniques. A system with a greater rating is a greater system.

In one among my research (Marie et al., 2021), I confirmed that just about 99% of the analysis papers in machine translation depend on the automated metric BLEU to guage translation high quality and rank techniques, whereas greater than 100 different metrics have been proposed over the last 12 years. Notice: I appeared solely at analysis papers printed from 2010 by the ACL. Probably many extra metrics have been proposed to guage machine translation.

Here’s a non-exhaustive checklist of 106 metrics proposed from 2010 to 2020 (click on on the metric identify to get the supply):

Noun-phrase chunking, SemPOS refinement, mNCD, RIBES, extended METEOR, Badger 2.0, ATEC 2.1, DCU-LFG, LRKB4, LRHB4, I-letter-BLEU, I-letter-recall, SVM-RANK,TERp, IQmt-DR, BEwT-E, Bkars, SEPIA, MEANT, AM-FM. AMBER, F15, MTeRater, MP4IBM1, ParseConf, ROSE, TINE, TESLA-CELAB, PORT, lexical cohesion, pFSM, pPDA, HyTER, SAGAN-STS, SIMPBLEU, SPEDE, TerrorCAT, BLOCKERRCATS, XENERRCATS, PosF, TESLA, LEPOR, ACTa, DEPREF, UMEANT, LogRefSS, discourse-based, XMEANT, BEER, SKL, AL-BLEU, LBLEU, APAC, RED-*, DiscoTK-*, ELEXR, LAYERED, Parmesan, tBLEU, UPC-IPA, UPC-STOUT, VERTa-*, pairwise neural, neural representation-based, ReVal, BS, LeBLEU, chrF, DPMF, Dreem, Ratatouille, UoW-LSTM, UPF-Colbat, USAAR-ZWICKEL, CharacTER, DepCheck, MPEDA, DTED, meaning features, BLEU2VEC_Sep, Ngram2vec, MEANT 2.0, UHH_TSKM, AutoDA, TreeAggreg, BLEND, HyTERA, RUSE, ITER, YiSi, BERTr, EED, WMDO, PReP, cross-lingual similarity+target language model, XLM+TLM, Prism, COMET, PARBLEU, PARCHRF, MEE, BLEURT, BAQ-*, OPEN-KIWI-*, BERT, mBERT, EQ-*

Most of those metrics have been proven to be higher than BLEU, however have by no means been used. Actually, solely 2 (1.8%) of those metrics, RIBES and chrF, have been utilized in greater than two analysis publications (among the many 700+ publications that I checked). Since 2010, essentially the most used metrics are metrics proposed earlier than 2010 (BLEU, TER, and METEOR):

Many of the metrics created after 2016 are neural metrics. They depend on neural networks and the newest ones even depend on the very talked-about pre-trained language fashions.

In distinction, conventional metrics printed earlier might be extra easy and cheaper to run. They continue to be extraordinarily in style for varied causes, and this recognition doesn’t appear to say no, at the very least in analysis.

Within the following sections, I introduce a number of metrics chosen in accordance with their recognition, their originality, or their correlation with human analysis.

Conventional metrics for machine translation analysis might be seen as metrics that consider the gap between two strings merely based mostly on the characters they comprise.

These two strings are the interpretation speculation and the reference translation. Notice: Sometimes, conventional metrics don’t exploit the supply textual content translated by the system.

WER (Phrase Error Price) was one essentially the most used of those metrics, and the ancestor of BLEU, earlier than BLEU took over within the early 2000’s.

Benefits:

• Low computational value: Most conventional metrics depend on the effectivity of string matching algorithms run at character and/or token ranges. Some metrics do must carry out some shifting of tokens which might be extra pricey, significantly for lengthy translations. Nonetheless, their computation is well parallelizable and doesn’t require a GPU.
• Explainable: Scores are often straightforward to compute by hand for small segments and thus facilitate the evaluation. Notice: “Explainable” doesn’t imply “interpretable”, i.e., we are able to precisely clarify how a metric rating is computed, however the rating alone can’t be interpreted because it often tells us nothing of the interpretation high quality.
• Language impartial: Besides some explicit metrics, the identical metric algorithms might be utilized independently of the language of the interpretation.

Disadvantages:

• Poor correlation with human judgments: That is their important drawback towards neural metrics. To get the most effective estimation of the standard of a translation, conventional metrics shouldn’t be used.
• Require explicit preprocessing: Aside from one metric (chrF), all the standard metric I current on this article requires the evaluated segments, and their reference translations, to be tokenized. The tokenizer isn’t embedded within the metric, i.e., it must be carried out by the person utilizing exterior instruments. The scores obtained are then depending on a selected tokenization that might not be reproducible.

BLEU

That is the preferred metric. It’s utilized by virtually 99% of the machine translation analysis publications.

I already offered BLEU in one among my previous article.

BLEU is a metric with many well-identified flaws.

What I didn’t talk about in my two articles about BLEU is the various variants of BLEU.

When studying analysis papers, you could discover metrics denoted BLEU-1, BLEU-2, BLEU-3, and so forth. The quantity after the hyphen is often the utmost size of the n-grams of tokens used to compute the rating.

As an illustration, BLEU-4 is a BLEU computed by taking {1,2,3,4}-grams of tokens under consideration. In different phrases, BLEU-4 is the standard BLEU computed in most machine translation papers, as initially proposed by Papineni et al. (2002).

BLEU is a metric that requires lots of statistics to be correct. It doesn’t work properly on brief textual content, and will even yield an error if computed on a translation that doesn’t match any 4-grams from the reference translation.

Since evaluating translation high quality at sentence degree could also be needed in some purposes or for evaluation, a variant denoted sentence BLEU, sBLEU, or generally BLEU+1 can be utilized. It avoids computational errors. There are a lot of variants of BLEU+1. The preferred ones are described by Chen and Cherry (2014).

As we’ll see with neural metrics, BLEU+1 has many higher alternate options and shouldn’t be used.

chrF(++)

chrF (Popović, 2015) is the second hottest metric for machine translation analysis.

It has been round since 2015 and has since been more and more utilized in machine translation publications.

It has been proven to higher correlate with human judgment than BLEU.

As well as, chrF is tokenization impartial. That is the one metric with this function that I do know of. Because it doesn’t require any prior customized tokenization by some exterior software, it is among the finest metrics to make sure the reproducibility of an analysis.

chrF solely depends on the characters. Areas are ignored by default.

chrF++ (Popović, 2017) is a variant of chrF that higher correlates with human analysis however at the price of its tokenization independence. Certainly, chrF++ exploits areas to take into consideration phrase order, therefore its higher correlation with human analysis.

I do strongly advocate the usage of chrF once I overview machine translation papers for conferences and journals to make an analysis extra reproducible, however not chrF++ as a consequence of its tokenization dependency.

Notice: Be cautious whenever you learn a analysis work utilizing chrF. Authors usually confuse chrF and chrF++. They might additionally cite the chrF paper when utilizing chrF++, and vice versa.

The original implementation of chrF by Maja Popović is on the market on github.

You may also discover an implementation in SacreBLEU (Apache 2.0 license).

RIBES

RIBES (Isozaki et al., 2010) is commonly utilized by the analysis neighborhood.

This metric was designed for “distant language pairs” with very completely different sentence buildings.

As an illustration, translating English into Japanese requires a big phrase reordering for the reason that verb in Japanese is positioned on the finish of the sentence whereas in English it’s often positioned earlier than the complement.

The authors of RIBES discovered that the metrics accessible at the moment, in 2010, weren’t sufficiently penalizing incorrect phrase order and thus proposed this new metric as a substitute.

An implementation of RIBES is available on Github (GNU Normal Public License V2.0).

METEOR

METEOR (Banerjee and Lavie, 2005) was first proposed in 2005 with the target of correcting a number of flaws of conventional metrics accessible at the moment.

As an illustration, BLEU solely counts actual token matches. It’s too strict since phrases should not rewarded by BLEU if they aren’t precisely the identical within the reference translation even when they’ve an analogous that means. As such, BLEU is blind to many legitimate translations.

METEOR partly corrects this flaw by introducing extra flexibility within the matching. Synonyms, phrase stems, and even paraphrases are all accepted as legitimate translations, successfully bettering the recall of the metric. The metric additionally implements a weighting mechanism to present extra significance, as an example, to a precise matching over a stem matching.

The metric is computed by the harmonic imply between recall and precision, with the particularity that the recall has the next weight than precision.

METEOR higher correlates with human analysis than BLEU, and has been improved a number of occasions till 2015. It’s nonetheless commonly used these days.

METEOR has an official webpage maintained by CMU which proposes the unique implementation of the metric (unknown license).

TER

TER (Snover et al., 2006) is principally used to guage the trouble it could take for a human translator to post-edit a translation.

Definition

Submit-editing in machine translation is the motion of correcting a machine translation output into an appropriate translation. Machine translation adopted by post-editing is a regular pipeline used within the translation trade to scale back translation value.

There are two well-known variants: TERp (Snover et al., 2009) and HTER (Snover et al., 2009, Specia and Farzindar, 2010).

TERp is TER augmented with a paraphrase database to enhance the recall of the metric and its correlation with human analysis. A match between the speculation and the reference is counted if a token, or one among its paraphrases, from the interpretation speculation is within the reference translation.

HTER, standing for “Human TER”, is a regular TER computed between machine translation speculation and its post-editing produced by a human. It may be used to guage the associated fee, a posteriori, of post-editing a selected translation.

CharacTER

The identify of the metric already provides some hints on the way it works: That is the TER metric utilized at character degree. Shift operations are carried out at phrase degree.

The edit distance obtained can be normalized by the size of the interpretation speculation.

CharacTER (Wang et al., 2016) has one of many highest correlation with human analysis among the many conventional metrics.

Nonetheless, it stays much less used than different metrics. I couldn’t discover any papers that used it just lately.

The implementation of characTER by its authors is on the market on Github (unknown license).

Neural metrics take a really completely different method from the standard metrics.

They estimate a translation high quality rating utilizing neural networks.

To the most effective of my data, ReVal, proposed in 2015, was the primary neural metric with the target of computing a translation high quality rating.

Since ReVal, new neural metrics are commonly proposed for evaluating machine translation.

The analysis effort in machine translation analysis is now virtually solely specializing in neural metrics.

But, as we’ll see, regardless of their superiority, neural metrics are removed from in style. Whereas neural metrics have been round for nearly 8 years, conventional metrics are nonetheless overwhelmingly most well-liked, at the very least by the analysis neighborhood (the scenario might be completely different within the machine translation trade).

Benefits:

• Good correlation with human analysis: Neural metrics are state-of-the-art for machine translation analysis
• No preprocessing required: That is primarily true for current neural metrics equivalent to COMET and BLEURT. The preprocessing, equivalent to tokenization, is completed internally and transparently by the metric, i.e., the customers don’t must care about it.
• Higher recall: Due to the exploitation of embeddings, neural metrics can reward translation even once they don’t precisely match the reference. As an illustration, a phrase that has a that means just like a phrase within the reference will likely be possible rewarded by the metric, in distinction to conventional metrics that may solely reward actual matches.
• Trainable: This can be a bonus in addition to a drawback. Most neural metrics have to be skilled. It is a bonus you probably have coaching knowledge on your particular use case. You’ll be able to fine-tune the metric to finest correlate with human judgments. Nevertheless, if you happen to don’t have the particular coaching knowledge, the correlation with human analysis will likely be removed from optimum.

Disadvantages:

• Excessive computational value: Neural metrics don’t require a GPU however are a lot sooner you probably have one. But, even with a GPU, they’re considerably slower than conventional metrics. Some metrics counting on giant language fashions equivalent to BLEURT and COMET additionally require a big quantity of reminiscence. Their excessive computational value additionally makes statistical significance testing extraordinarily pricey.
• Unexplainable: Understanding why a neural metric yields a selected rating is almost not possible for the reason that neural mannequin behind it usually leverages thousands and thousands or billions of parameters. Enhancing the explainability of neural fashions is a really lively analysis space.
• Troublesome to keep up: Older implementations of neural metrics don’t work anymore in the event that they weren’t correctly maintained. That is primarily because of the modifications in nVidia CUDA and/or frameworks equivalent to (py)Torch and Tensorflow. Probably, the present model of the neural metrics we use at the moment received’t work in 10 years.
• Not reproducible: Neural metrics often include many extra hyperparameters than conventional metrics. These are largely underspecified within the scientific publications utilizing them. Subsequently, reproducing a selected rating for a selected dataset is usually not possible.

ReVal

To the most effective of my data, ReVal (Gupta et al., 2015) is the primary neural metric proposed to guage machine translation high quality.

ReVal was a big enchancment over conventional metrics with a considerably higher correlation with human analysis.

The metric is predicated on an LSTM and may be very easy, however has by no means been utilized in machine translation analysis so far as I do know.

It’s now outperformed by more moderen metrics.

In case you are to know the way it works, you may nonetheless discover ReVal’s original implementation on Github (GNU Normal Public License V2.0).

YiSi

YiSi (Chi-kiu Lo, 2019) is a really versatile metric. It primarily exploits an embedding mannequin however might be augmented with varied assets equivalent to a semantic parser, a big language mannequin (BERT), and even options from the supply textual content and supply language.

Utilizing all these choices could make it pretty advanced and reduces its scope to a couple language pairs. Furthermore, the positive factors when it comes to correlation with human judgments when utilizing all these choices should not apparent.

Nonetheless, the metric itself, utilizing simply the unique embedding mannequin, exhibits an excellent correlation with human analysis.

The creator confirmed that for evaluating English translations YiSi considerably outperforms conventional metrics.

The unique implementation of YiSi is publicly available on Github (MIT license).

BERTScore

BERTScore (Zhang et al., 2020) exploits the contextual embeddings of BERT for every token within the evaluated sentence and compares them with the token embeddings of the reference.

It really works as illustrated under:

It is among the first metrics to undertake a big language mannequin for analysis. It wasn’t proposed particularly for machine translation however fairly for any language era process.

BERTScore is essentially the most used neural metric in machine translation analysis.

A BERTScore implementation is on the market on Github (MIT license).

BLEURT

BLEURT (Sellam et al., 2020) is one other metric counting on BERT however that may be particularly skilled for machine translation analysis.

Extra exactly, it’s a BERT mannequin fine-tuned on artificial knowledge which might be sentences from Wikipedia paired with their random perturbations of various varieties: Notice: This step is confusedly denoted “pre-training” by the authors (see word 3 within the paper) but it surely really comes after the unique pre-training of BERT.

• Masked phrase (as within the unique BERT)
• Dropped phrase
• Backtranslation (i.e., sentences generated by a machine translation system)

Every sentence pair is evaluated throughout coaching with a number of losses. A few of these losses are computed with analysis metrics:

Lastly, in a second section, BLEURT is fine-tuned on translations and their score offered by people.

Intuitively, due to the usage of artificial knowledge which will resemble machine translation errors or outputs, BLEURT is far more sturdy to high quality and area drifts than BERTScore.

Furthermore, since BLEURT exploits a mix of metric as “pre-training indicators”, it’s intuitively higher than every one among these metrics, together with BERTScore.

Nevertheless, BLEURT may be very pricey to coach. I’m solely conscious of BLEURT checkpoints launched by Google. Notice: In case you are conscious of different fashions, please let me know within the feedback.

The primary model was solely skilled for English, however the newer model, denoted BLEURT-20, now contains 19 extra languages. Both BLEURT versions are available in the same repository.

Prism

Of their work proposing Prism, Thompson and Post (2019) intuitively argue that machine translation and paraphrasing analysis are very related duties. Their solely distinction is that the supply language just isn’t the identical.

Certainly, with paraphrasing, the target is to generate a brand new sentence A’, given a sentence A, with A and A’ having the identical that means. Assessing how shut A and A’ is similar to assessing how a translation speculation is near a given reference translation. In different phrases, is the interpretation speculation an excellent paraphrase of the reference translation.

Prism is a neural metric skilled on a big multilingual parallel dataset via a multilingual neural machine translation framework.

Then, at inference time, the skilled mannequin is used as a zero-shot paraphraser to attain the similarity between a supply textual content (the interpretation speculation) and the goal textual content (the reference translation) which might be each in the identical language.

The principle benefit of this method is that Prism doesn’t want any human analysis coaching knowledge nor any paraphrasing coaching knowledge. The one requirement is to have parallel knowledge for the languages you intend to guage.

Whereas Prism is unique, handy to coach, and appears to outperform most different metrics (together with BLEURT), I couldn’t discover any machine translation analysis publication utilizing it.

The unique implementation of Prism is publicly available on Github (MIT license).

COMET

COMET (Rei et al., 2020) is a extra supervised method additionally based mostly on a big language mannequin. The authors chosen XLM-RoBERTa however point out that different fashions equivalent to BERT might additionally work with their method.

In distinction to most different metrics, COMET exploits the supply sentence. The massive language mannequin is thus fine-tuned on a triplet {translated supply sentence, translation speculation, reference translation}.

The metric is skilled utilizing human scores (the identical ones utilized by BLEURT).

COMET is far more easy to coach than BLEURT because it doesn’t require the era and the scoring of artificial knowledge.

COMET is on the market in lots of variations, together with distilled fashions (COMETHINO) which have a a lot smaller reminiscence footprint.

The released implementation of COMET (Apache license 2.0) additionally features a software to effectively carry out statistical significance testing.

Machine translation analysis is a very lively analysis space. Neural metrics are getting higher and extra environment friendly yearly.

But, conventional metrics equivalent to BLEU stay the favorites of machine translation practitioners, primarily by habits.

In 2022, the Conference on Machine Translation (WMT22) published a ranking of evaluation metrics in accordance with their correlation with human analysis, together with metrics I offered on this article:

COMET and BLEURT rank on the high whereas BLEU seems on the backside. Curiously, you may as well discover on this desk that there are some metrics that I didn’t write about on this article. A few of them, equivalent to MetricX XXL, are undocumented.

Regardless of having numerous higher alternate options, BLEU stays by far essentially the most used metric, at the very least in machine translation analysis.

Private suggestions:

Once I overview scientific papers for conferences and journals, I all the time advocate the next to the authors who solely use BLEU for machine translation analysis:

• Add the outcomes for at the very least one neural metric equivalent to COMET or BLEURT, if the language pair is roofed by these metrics.
• Add the outcomes for chrF (not chrF++). Whereas chrF just isn’t state-of-the-art, it’s considerably higher than BLEU, yield scores which might be simply reproducible, and can be utilized for diagnostic functions.