RNN language model with word clustering and class-based output layer

Yongzhe Shi; Wei Qiang Zhang; Jia Liu; Michael T. Johnson

doi:10.1186/1687-4722-2013-22

RNN language model with word clustering and class-based output layer

Yongzhe Shi
, Wei Qiang Zhang
, Jia Liu
, Michael T. Johnson

Research output: Contribution to journal › Article › peer-review

28 Scopus citations

Abstract

The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

Original language	English
Article number	22
Journal	Eurasip Journal on Audio, Speech, and Music Processing
Volume	2013
Issue number	1
DOIs	https://doi.org/10.1186/1687-4722-2013-22
State	Published - 2013

Bibliographical note

Funding Information:
This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

Funding

This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

Funders	Funder number
Natural Science Foundation of Beijing Municipality	KZ201110005005
Natural Science Foundation of Beijing Municipality
National Natural Science Foundation of China (NSFC)	61273268, 61005019, 90920302
National Natural Science Foundation of China (NSFC)

Keywords

Brown word clustering
RNN language model
Speech recognition

ASJC Scopus subject areas

Acoustics and Ultrasonics
Electrical and Electronic Engineering

Access to Document

10.1186/1687-4722-2013-22

Cite this

@article{c26b8f5381764e278b3d61358ff7c023,

title = "RNN language model with word clustering and class-based output layer",

abstract = "The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.",

keywords = "Brown word clustering, RNN language model, Speech recognition",

author = "Yongzhe Shi and Zhang, \{Wei Qiang\} and Jia Liu and Johnson, \{Michael T.\}",

note = "Funding Information: This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005. ",

year = "2013",

doi = "10.1186/1687-4722-2013-22",

language = "English",

volume = "2013",

number = "1",

}

TY - JOUR

T1 - RNN language model with word clustering and class-based output layer

AU - Shi, Yongzhe

AU - Zhang, Wei Qiang

AU - Liu, Jia

AU - Johnson, Michael T.

N1 - Funding Information: This work was supported by the National Natural Science Foundation of China under grant nos. 61273268, 61005019 and 90920302, and in part by Beijing Natural Science Foundation Program under grant no. KZ201110005005.

PY - 2013

Y1 - 2013

N2 - The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

AB - The recurrent neural network language model (RNNLM) has shown significant promise for statistical language modeling. In this work, a new class-based output layer method is introduced to further improve the RNNLM. In this method, word class information is incorporated into the output layer by utilizing the Brown clustering algorithm to estimate a class-based language model. Experimental results show that the new output layer with word clustering not only improves the convergence obviously but also reduces the perplexity and word error rate in large vocabulary continuous speech recognition.

KW - Brown word clustering

KW - RNN language model

KW - Speech recognition

UR - https://www.scopus.com/pages/publications/84887089323

UR - https://www.scopus.com/pages/publications/84887089323#tab=citedBy

U2 - 10.1186/1687-4722-2013-22

DO - 10.1186/1687-4722-2013-22

M3 - Article

AN - SCOPUS:84887089323

SN - 1687-4714

VL - 2013

JO - Eurasip Journal on Audio, Speech, and Music Processing

JF - Eurasip Journal on Audio, Speech, and Music Processing

IS - 1

M1 - 22

ER -

RNN language model with word clustering and class-based output layer

Abstract

Bibliographical note

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this