Comparing the Accuracy of ACE and WER Caption Metrics When Applied to Live Television Captioning

The development of caption metrics is relatively new in the accessibility research community. However, little work has been done comparing the effectiveness of newly developed caption metrics. More specifically, in low accuracy settings such as live television, where users report the most difficulty using captions. Through a user study with fifteen participants, we compared two caption metrics systems, Word Error Rate (WER) and Automated-Caption Evaluation (ACE), for their accuracy in evaluating caption quality in live television. We compared human-perceived quality statistics with each caption metric’s data. Analysis of the correlation between human statistics and each caption metric found that WER had a slightly higher correlation with participants. We found that ACE was more sensitive to errors that WER, and penalized captions more than participants. However, the difference in performance between WER and ACE was not statistically significant, and neither WER nor ACE are optimized for use with live television captioning. Future work should explore how caption metrics could be better optimized for use with live television.