Statistical Language Models Based on Neural Networks

VYSOK

´

E U

ˇ

CEN

´

I TECHNICK

´

E V BRN

BRNO UNIVERSITY OF TECHNOLOGY

FAKULTA INFORMA

CN

USTAV PO

C

´

ITA

COV

E GRAFIKY A MULTIM

´

EDI

´

I

ˇ

´

PHD THESIS

AUTOR PR

ACE Ing. TOM

ˇ

S MIKOLOV

AUTHOR

BRNO 2012

VYSOK

´

E U

CEN

´

I TECHNICK

ˇ

E

USTAV PO

ITA

ˇ

COV

E GRAFIKY A MULTIM

EDI

I

FACULTY OF INFORMATION TECHNOLOGY

DEPARTMENT OF COMPUTER GRAPHICS AND MULTIMEDIA

STATISTICK

´

E JAZYKOV

NA NEURONOV

YCH S

´

IT

´

ICH

STATISTICAL LANGUAGE MODELS BASED ON NEURAL NETWORKS

DISERTA

CN

ˇ

AUTHOR

VEDOUC

´

I PR

´

ACE Doc. Dr. Ing. JAN

ˇ

CERNOCK

´

Y

SUPERVISOR

BRNO 2012

aplikace Google Translate). Tradiˇcn´ı techniky pro odhad tˇechto model˚u jsou zaloˇzeny

na tzv. N-gramech. Navzdory zn´am´ym nedostatk˚um tˇechto technik a obrovsk´emu ´usil´ı

v´yzkumn´ych skupin napˇr´ıˇc mnoha oblastmi (rozpozn´av´an´ı ˇreˇci, automatick´y pˇreklad, neu-

roscience, umˇel´a inteligence, zpracov´an´ı pˇrirozen´eho jazyka, komprese dat, psychologie

atd.), N -gramy v podstatˇe z˚ustaly nej´uspˇeˇsnˇejˇs´ı technikou. C´ılem t´eto pr´ace je prezen-

tace nˇekolika architektur jazykov´ych model˚u zaloˇzen´ych na neuronov´ych s´ıt´ıch. Aˇckoliv

jsou tyto modely v´ypoˇcetnˇe n´aroˇcnˇejˇs´ı neˇz N-gramov´e modely, s technikami vyvinut´ymi v

t´eto pr´aci je moˇzn´e jejich efektivn´ı pouˇzit´ı v re´aln´ych aplikac´ıch. Dosaˇzen´e sn´ıˇzen´ı poˇctu

chyb pˇri rozpozn´av´an´ı ˇreˇci oproti nejlepˇs´ım N -gramov´ym model˚um dosahuje 20%. Model

zaloˇzen´y na rekurentn´ı neurovov´e s´ıti dosahuje nejlepˇs´ıch publikovan´ych v´ysledk˚u na velmi

nities across many fields (speech recognition, machine translation, neuroscience, artificial

intelligence, natural language processing, data compression, psychology etc.), N-grams

remained basically the state-of-the-art. The goal of this thesis is to present various archi-

tectures of language models that are based on artificial neural networks. Although these

models are computationally more expensive than N -gram models, with the presented

techniques it is possible to apply them to state-of-the-art systems efficiently. Achieved

reductions of word error rate of speech recognition systems are up to 20%, against state-

of-the-art N -gram model. The presented recurrent neural network based model achieves

the best published performance on well-known Penn Treebank setup.

Kl´ıˇcov´a slova

Citace

Tom´aˇs Mikolov: Statistical Language Models Based on Neural Networks, disertaˇcn´ı pr´ace,

Brno, FIT VUT v Brnˇe, 2012

Statistical Language Models Based on Neural Net-

works

Prohl´aˇsen´ı

Prohlaˇsuji, ˇze jsem tuto disertaˇcn´ı pr´aci vypracoval samostatnˇe pod veden´ım Doc. Dr.

Ing. Jana

Nˇekter´e experimenty byly provedeny ve spolupr´aci s dalˇs´ımi ˇcleny skupiny Speech@FIT,

ˇ

proaches to standard problems, for his support and constructive criticism of my work,

and for his ability to quickly organize everything related to my studies. I am grateful to

Luk´aˇs Burget for many advices he gave me about speech recognition systems, for long

discussions about many technical details and for his open-minded approach to research.

I would also like to thank all members of Speech@FIT group for cooperation, especially

Stefan Kombrink, Oldˇrich Plchot, Martin Karafi´at, Ondˇrej Glembek and Jiˇr´ı Kopeck´y.

It was great experience for me to visit Johns Hopkins University during my studies, and I

am grateful to Frederick Jelinek and Sanjeev Khudanpur for granting me this opportunity.

I always enjoyed discussions with Sanjeev, who was my mentor during my stay there. I

personally.

© Tom´aˇs Mikolov, 2012.

Tato pr´ace vznikla jako ˇskoln´ı d´ılo na Vysok´em uˇcen´ı technick´em v Brnˇe, Fakultˇe in-

formaˇcn´ıch technologi´ı. Pr´ace je chr´anˇena autorsk´ym z´akonem a jej´ı uˇzit´ı bez udˇelen´ı

opr´avnˇen´ı autorem je nez´akonn´e, s v´yjimkou z´akonem definovan´ych pˇr´ıpad˚u.