Lost in the machine: can AI handle dialects in police interviews?

The study tested ASR transcription performance using two British English accents: Standard Southern British English and West Yorkshire English. Three commercially available ASR transcription systems were tested, with different audio qualities.

The analysis assessed different error types:

• insertions: when the system adds in an extra word;

• deletions: when the system leaves a word out; and

• substitutions: when the system incorrectly swaps in a new word rather than transcribing the word that is spoken in the audio recording. 

A higher number of errors were produced for the West Yorkshire English speech compared with the Standard Southern British English speech. The majority of errors for the West Yorkshire speakers were substitution errors. These are potentially more damaging in high-stakes or forensic contexts because they change the meaning of words and could prime listeners or readers into thinking that something was said which never actually appeared in the original audio file. However, many of the errors were quite easy to identify, and therefore should be picked up by a human checker.