Evaluation Metrics

1. Accuracy

Accuracy refers to the average accuracy of the model across all evaluation instances. The concept of correctness may vary from case to case, so we present the main accuracy measures considered in the measurement, the scenarios in which these measures are used, and the associated formal definitions.

1.1 Character Error Rate

Character Error Rate, CER measures the character-level difference between generated text and reference text. CER calculates the proportion of characters in the generated text that do not match the reference text (including deletion, insertion, and substitution errors). The lower the CER value, the closer the generated text is to the reference text and the better the performance. CER is a ratio between 0 and 1, usually expressed as a percentage.

1.2 Word Error Rate

Word Error Rate, WER measures the word-level difference between generated text and reference text. WER calculates the proportion of words in the generated text that do not match the reference text (including deletion, insertion, and substitution errors). The lower the WER value, the closer the generated text is to the reference text and the better the performance. WER is a ratio between 0 and 1, usually expressed as a percentage.