Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR

Mohammad Soleymanpour; Michael T. Johnson; Jeffrey Berry

doi:10.1109/SpeD53181.2021.9587372

Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR

Mohammad Soleymanpour, Michael T. Johnson, Jeffrey Berry

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

End-to-end speech recognition systems are effective, but in order to train an end-to-end model, a large amount of training data is needed. For applications such as dysarthric speech recognition, we do not have sufficient data. In this paper, we propose a specialized data augmentation approach to enhance the performance of an end-to-end dysarthric ASR based on sub-word models. The proposed approach contains two methods, including prosodic transformation and time-feature masking. Prosodic transformation modifies the speaking rate and pitch of normal speech to control prosodic characteristics such as loudness, intonation, and rhythm. Using time and feature masking, we apply a mask to the Mel Frequency Cepstral Coefficients (MFCC) for robustness-focused augmentation. Results show that augmenting normal speech with prosodic transformation plus masking decreases CER by 5.4% and WER by 5.6%, and the further addition of dysarthric speech masking decreases CER by 11.3% and WER by 11.4%.

Original language	English
Title of host publication	2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021
Pages	42-46
Number of pages	5
ISBN (Electronic)	9781665427869
DOIs	https://doi.org/10.1109/SpeD53181.2021.9587372
State	Published - 2021
Event	11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021 - Virtual, Bucharest, Romania Duration: Oct 13 2021 → Oct 15 2021

Publication series

Name	2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021

Conference

Conference	11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021
Country/Territory	Romania
City	Virtual, Bucharest
Period	10/13/21 → 10/15/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Keywords

Data augmentation
Dysarthria
Dysarthric ASR
Speech recognition
Subwordmodel

ASJC Scopus subject areas

Language and Linguistics
Human-Computer Interaction
Signal Processing
Safety, Risk, Reliability and Quality
Communication

Access to Document

10.1109/SpeD53181.2021.9587372

Cite this

Soleymanpour, M., Johnson, M. T., & Berry, J. (2021). Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR. In 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021 (pp. 42-46). (2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021). https://doi.org/10.1109/SpeD53181.2021.9587372

Soleymanpour, Mohammad ; Johnson, Michael T. ; Berry, Jeffrey. / Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR. 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021. 2021. pp. 42-46 (2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021).

@inproceedings{f791e6865d93432e961e82287f59c3c6,

title = "Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR",

abstract = "End-to-end speech recognition systems are effective, but in order to train an end-to-end model, a large amount of training data is needed. For applications such as dysarthric speech recognition, we do not have sufficient data. In this paper, we propose a specialized data augmentation approach to enhance the performance of an end-to-end dysarthric ASR based on sub-word models. The proposed approach contains two methods, including prosodic transformation and time-feature masking. Prosodic transformation modifies the speaking rate and pitch of normal speech to control prosodic characteristics such as loudness, intonation, and rhythm. Using time and feature masking, we apply a mask to the Mel Frequency Cepstral Coefficients (MFCC) for robustness-focused augmentation. Results show that augmenting normal speech with prosodic transformation plus masking decreases CER by 5.4% and WER by 5.6%, and the further addition of dysarthric speech masking decreases CER by 11.3% and WER by 11.4%.",

keywords = "Data augmentation, Dysarthria, Dysarthric ASR, Speech recognition, Subwordmodel",

author = "Mohammad Soleymanpour and Johnson, {Michael T.} and Jeffrey Berry",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021 ; Conference date: 13-10-2021 Through 15-10-2021",

year = "2021",

doi = "10.1109/SpeD53181.2021.9587372",

language = "English",

series = "2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021",

pages = "42--46",

booktitle = "2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021",

}

Soleymanpour, M, Johnson, MT & Berry, J 2021, Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR. in 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021. 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021, pp. 42-46, 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021, Virtual, Bucharest, Romania, 10/13/21. https://doi.org/10.1109/SpeD53181.2021.9587372

Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR. / Soleymanpour, Mohammad; Johnson, Michael T.; Berry, Jeffrey.
2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021. 2021. p. 42-46 (2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR

AU - Soleymanpour, Mohammad

AU - Johnson, Michael T.

AU - Berry, Jeffrey

PY - 2021

Y1 - 2021

N2 - End-to-end speech recognition systems are effective, but in order to train an end-to-end model, a large amount of training data is needed. For applications such as dysarthric speech recognition, we do not have sufficient data. In this paper, we propose a specialized data augmentation approach to enhance the performance of an end-to-end dysarthric ASR based on sub-word models. The proposed approach contains two methods, including prosodic transformation and time-feature masking. Prosodic transformation modifies the speaking rate and pitch of normal speech to control prosodic characteristics such as loudness, intonation, and rhythm. Using time and feature masking, we apply a mask to the Mel Frequency Cepstral Coefficients (MFCC) for robustness-focused augmentation. Results show that augmenting normal speech with prosodic transformation plus masking decreases CER by 5.4% and WER by 5.6%, and the further addition of dysarthric speech masking decreases CER by 11.3% and WER by 11.4%.

AB - End-to-end speech recognition systems are effective, but in order to train an end-to-end model, a large amount of training data is needed. For applications such as dysarthric speech recognition, we do not have sufficient data. In this paper, we propose a specialized data augmentation approach to enhance the performance of an end-to-end dysarthric ASR based on sub-word models. The proposed approach contains two methods, including prosodic transformation and time-feature masking. Prosodic transformation modifies the speaking rate and pitch of normal speech to control prosodic characteristics such as loudness, intonation, and rhythm. Using time and feature masking, we apply a mask to the Mel Frequency Cepstral Coefficients (MFCC) for robustness-focused augmentation. Results show that augmenting normal speech with prosodic transformation plus masking decreases CER by 5.4% and WER by 5.6%, and the further addition of dysarthric speech masking decreases CER by 11.3% and WER by 11.4%.

KW - Data augmentation

KW - Dysarthria

KW - Dysarthric ASR

KW - Speech recognition

KW - Subwordmodel

UR - http://www.scopus.com/inward/record.url?scp=85125639209&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85125639209&partnerID=8YFLogxK

U2 - 10.1109/SpeD53181.2021.9587372

DO - 10.1109/SpeD53181.2021.9587372

M3 - Conference contribution

AN - SCOPUS:85125639209

T3 - 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021

SP - 42

EP - 46

BT - 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021

T2 - 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021

Y2 - 13 October 2021 through 15 October 2021

ER -

Soleymanpour M, Johnson MT, Berry J. Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR. In 2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021. 2021. p. 42-46. (2021 11th International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2021). doi: 10.1109/SpeD53181.2021.9587372

Dysarthric speech augmentation using prosodic transformation and masking for subword end-to-end ASR

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this