Caption Accuracy Metrics

The WGBH National Center for Accessible Media (NCAM) has launched a project to develop a tool that can measure the accuracy of live captioning on television and elsewhere. This is in response to consumer complaints about the quality of captions. The press release describing the project and what it hopes to accomplish are pasted here. Head up, chin out, eyes forward, double thumbs up…onward, NCAM, ever onward!

Real-time captioned news is a lifeline service for people who are deaf or hard of hearing, providing critical information about their local communities, national events and emergencies. Captioning mandates designed to provide equal access to television have resulted in rapid growth of the caption industry, but a shortage of skilled real-time stenocaptioners, and the downward pressure on rates by program providers, has made the low quality of live captioning of news broadcasts a growing issue.

Disability organizations have filed complaints and a formal petition with the Federal Communications Commission (FCC) which reflects frustration with chronic problems related to live captioning quality, transmission errors, and lack of industry response to their concerns. However, without a common means of measuring accuracy and quality, the FCC, consumers and broadcasters have no efficient method of tracking and improving stenocaption accuracy performance.

The WGBH National Center for Accessible Media is utilizing language-processing tools to develop a prototype automated caption accuracy assessment system for real-time captions for live news programming or classroom-based communication access realtime translation (CART) captioning. We are researching whether text-based data mining and automatic speech recognition technologies can produce meaningful data about stenocaption accuracy that meets the need for caption performance metrics.

Prototypes will be reviewed by major stakeholders at Technical Review Meetings. Advisors include the National Institute of Standards and Technology, the Massachusetts Institute of Technology, Gallaudet University and the National Technical Institute for the Deaf.

Iterative tests and modifications within major stenocaption and broadcast operations facilities will provide real-world assessments of the system’s ability to produce meaningful caption accuracy metrics.

A reliable performance measurement tool that can analyze the quality of real-time captioning, developed with input from industry leaders, deaf education experts, and the National Institute of Standards and Technology will provide Congress and the FCC with much-needed independently verified data to establish caption accuracy requirements. This will greatly improve the ability of the television community to monitor and maintain the quality of live captioning they offer to viewers who are deaf or hard of hearing and ease the current burden on caption viewers to document and advocate for comprehensible captions to ensure they have equal access to important national and local information.

Deliverables include:

  • Publication of experimental ontology of caption error types;
  • Publication of research into error capture capabilities of text mining software agents, customized with rules and classifications derived from stenocaption ontology; and
  • Prototype software application that provides a technical framework to utilize language processing tools to conduct ontology-based detection, ranking and reporting of stenocaption errors.


One response to this post.

  1. This is good news, it is probably overdue. I wonder if anyone in the UK is working on such a project either with WGBH or not. It would be great if such technology could be implemented worldwide.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: