Automatic Pronunciation Feedback

NetProF allows users to listen to native speaker recording of words and phrases. After repeating the items back, the user receives scores and color-coded feedback of their pronunciation at the phrase, word, and phoneme level for individual speech sounds.

Each year, the U.S. government trains thousands of soldiers at the Defense Language Institute Foreign Language Center (DLIFLC) in languages critical to national security. Many of these languages are not widely studied in the United States, and students benefit from being able to practice proper pronunciation. There is a growing need for computer-based tools that can augment classroom training and provide feedback to students at their own pace. To address this need, Lincoln Laboratory developed a new tool to help students improve their pronunciation and general language fluency.

NetProF allows students to listen to native speaker recordings of vocabulary items and to record themselves speaking the items back. They then receive scores of how well they spoke the overall item, each word in the item, and each phoneme (distinct sound) within a word. This pronunciation feedback is made possible by Lincoln Laboratory’s automatic speech recognition technology: we trained neural network–based phoneme recognition models using in-house audio collections.

High-fidelity custom dictionaries provide precise word pronunciations, and our decoder provides scores in typically less than two seconds. NetProF’s vocabulary parallels DLIFLC’s classroom content, grouping vocabulary items into units and chapters. The audio content includes male and female reference recordings at normal and slow speeds. Each item is accompanied by an example sentence with the item used in context. Score feedback is provided via intuitive color highlighting. The student can replay their speech and see a “follow the bouncing ball” dynamic display that highlights each word and phoneme as it plays. An overall summary display shows which phonemes the student has trouble pronouncing and examples of words where these phonemes appear. The student can also touch a word and compare their speech against a native speaker’s pronunciation.

NetProF was fully transitioned to DLIFLC in 2019 and is in operational use for Modern Standard Arabic, English, Dari, Pashto, Farsi, Urdu, Mandarin Chinese, Korean, Russian, Spanish, French, Portuguese, German, and Tagalog.  MIT Lincoln Laboratory continues to develop specialized human language technology applications for U.S. government sponsors.