Skip to content
Neattrim

Auto-caption accuracy by language: a test method

· Editorial team · Updated

Do it now
Subtitle generator
Auto subtitles / captions for a video.

This page has no accuracy numbers for auto captions: we have not run this test. What it gives is the method: a fixed set of recordings per language, a human reference transcript, and word error rate (character error rate for some scripts). You can run it on your own recordings today.

Status of this page

This is a method page. The method is written down; no data exists yet, so there is no table of numbers here and none is implied. If results are ever added, they come with their data file and a new modified date.

What the test measures

For each language, the test measures how far the automatic transcript is from a transcript written and checked by a person. The standard measure is word error rate (WER). It counts the words the system got wrong, as substitutions, deletions and insertions, and divides by the number of words in the reference.

WER = (substitutions + deletions + insertions) ÷ words in the reference

A lower number is better. Zero means a perfect match. The number can go above 100% if the system adds many extra words.

Why one number per language is not enough

Accuracy depends on much more than the language. Score each condition separately inside each language, because a single score hides the cases that matter to creators:

  • Clean speech from a close microphone.
  • Speech with room noise or music under it.
  • Phone recordings made at arm's length.
  • Fast, casual speech compared with read speech.
  • Speakers with different accents.
  • Code-switching, where a speaker mixes two languages in one sentence, such as Hindi and English.

How to build the test set

  1. Choose the languages you need captions in.

  2. For each language, collect recordings you have the right to use, in the conditions listed above. Do not use videos taken from other people's channels.

  3. A fluent speaker writes the reference transcript. A second person checks it. Spelling and number conventions are fixed in advance, for example whether "20" is written as a digit or as a word.

  4. Run the same recordings through the subtitle generator with the spoken language set to the right one, and again with language detection on.

  5. Normalise both texts the same way before scoring: lowercase where the script has case, strip punctuation, and apply the number rule from step 3.

  6. Score WER per recording and per condition. For languages written without spaces between words, such as Japanese and Chinese, score character error rate instead.

What a complete run records

A run is only useful to others when it keeps these records together:

  • The results table by language and condition, with the date of the run and the version of the tool.
  • A data file with the per-recording scores.
  • The list of recordings by id, with language, length and condition, though not the audio itself if we lack the right to share it.
  • The scoring script, so anyone can reproduce the numbers.

Test it yourself

Take one minute of your own speech, write the transcript by hand, run the clip through the subtitle generator, and count the wrong words. Always read the captions through once before you publish a video. If you caption for accessibility, see captions and accessibility: the WCAG basics.

Frequently asked questions

Why do you not publish numbers from other studies?

Numbers from another system, another test set or another year do not describe this tool. This page shows only numbers we measured ourselves, with a date, and today it shows none.

What is a good word error rate?

There is no single answer. A clean, read recording scores much lower than a noisy phone call. Compare scores only when the recordings are alike.

Does the tool support my language?

The subtitle generator page lists the spoken languages it handles. Setting the language yourself is usually more reliable than leaving detection on.

How do I test Hinglish?

Treat code-switched speech as its own condition, and fix in advance whether English words are written in Latin script or in Devanagari, because the two give different scores. See Hinglish captions: why and how.

Are there results for this tool?

No. We have not run this test, so this page reports no accuracy figures for the subtitle generator.

Related guides