RNN language model with word clustering and class-based output layer

Yongzhe Shi; Wei Qiang Zhang; Jia Liu; Michael T. Johnson

doi:10.1186/1687-4722-2013-22

RNN language model with word clustering and class-based output layer

Yongzhe Shi, Wei Qiang Zhang, Jia Liu, Michael T. Johnson

Producción científica: Article › revisión exhaustiva

28 Citas (Scopus)

Resumen

The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

Idioma original	English
Número de artículo	22
Publicación	Eurasip Journal on Audio, Speech, and Music Processing
Volumen	2013
N.º	1
DOI	https://doi.org/10.1186/1687-4722-2013-22
Estado	Published - 2013

Nota bibliográfica

Funding Information:
This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

Financiación

This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

Financiadores	Número del financiador
Natural Science Foundation of Beijing Municipality	KZ201110005005
Natural Science Foundation of Beijing Municipality
National Natural Science Foundation of China (NSFC)	61273268, 61005019, 90920302
National Natural Science Foundation of China (NSFC)

ASJC Scopus subject areas

Acoustics and Ultrasonics
Electrical and Electronic Engineering

Acceder al documento

10.1186/1687-4722-2013-22

Otros archivos y enlaces

Citar esto

@article{c26b8f5381764e278b3d61358ff7c023,

title = "RNN language model with word clustering and class-based output layer",

abstract = "The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.",

keywords = "Brown word clustering, RNN language model, Speech recognition",

author = "Yongzhe Shi and Zhang, \{Wei Qiang\} and Jia Liu and Johnson, \{Michael T.\}",

note = "Funding Information: This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005. ",

year = "2013",

doi = "10.1186/1687-4722-2013-22",

language = "English",

volume = "2013",

number = "1",

}

TY - JOUR

T1 - RNN language model with word clustering and class-based output layer

AU - Shi, Yongzhe

AU - Zhang, Wei Qiang

AU - Liu, Jia

AU - Johnson, Michael T.

N1 - Funding Information: This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

PY - 2013

Y1 - 2013

N2 - The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

AB - The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

KW - Brown word clustering

KW - RNN language model

KW - Speech recognition

UR - http://www.scopus.com/inward/record.url?scp=84887089323&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84887089323&partnerID=8YFLogxK

U2 - 10.1186/1687-4722-2013-22

DO - 10.1186/1687-4722-2013-22

M3 - Article

AN - SCOPUS:84887089323

SN - 1687-4714

VL - 2013

JO - Eurasip Journal on Audio, Speech, and Music Processing

JF - Eurasip Journal on Audio, Speech, and Music Processing

IS - 1

M1 - 22

ER -

RNN language model with word clustering and class-based output layer

Resumen

Nota bibliográfica

Financiación

ASJC Scopus subject areas

Acceder al documento

Otros archivos y enlaces

Huella

Citar esto