Frengly.com engine uses its own method to measure translation quality called Machine Translation Quality Index.

The calculation is really simple:

The formula above is a sum of segments evaluations divided by number of words.
Segments are blocks of words (usually not longer than 4) in the source text.
Let's see the example (EN→FR):

So how exactly did we come up with the score?

Try to hover your mouse on the output translation, you will notice that every segment has been rated with a score.

And here we have reached the most important part - segment rating. Here it is:

- 1 for every word inside a human translated segment

- 1 for every word inside machine translated segment of size > 1

- 0.80 for a machine translated segment of size 1 (word) which length is < 4

- 0.75 for a machine translated segment of size 1 (word) which length is ≥ 4

- 0 for missing translation

What is the logic behind this scoring?

Marks defined in the scoring grid above have been defined commonly for all languages as an average values. It implies certain accuracy margin that usually ranges from 2% to 10%. The margin is an acceptable result of compromise between accuracy and process simplification.

Two remarks can be concluded from the scoring grid: first - MTQI promotes the human translations over machine translations (this is natural for hybrid translation engines), and the second - when evaluating single word translations (segment of a size 1), longer words have better score, as the engine assumes that short words are usually pronouns and prepositions, which should be translated along with other words.

The prime idea behind the MTQI methodology is to make sure that top scores are reserved for accurate translations only, so as the translator engine is able to monitor its quality and progressively improve by removing weak points.

download the article Measurement of translation quality