LearningPhraseRepresentationsusingRNNEncoder

需积分: 46 90 浏览量 2018-03-07 23:19:54 上传评论收藏 1.09MB PDF 举报

### 使用RNN编码器-解码器学习短语表示 #### 摘要与引言本文介绍了一种新颖的神经网络模型——RNN编码器-解码器（Recurrent Neural Network Encoder-Decoder），该模型由两个循环神经网络组成：一个用于将一系列符号编码为固定长度的向量表示，另一个则将此向量表示解码为另一系列符号。这种编码器和解码器联合训练以最大化给定源序列的目标序列条件概率。实证研究表明，在现有的对数线性模型中引入由RNN编码器-解码器计算的短语对条件概率作为附加特征后，统计机器翻译系统的性能得到了显著提升。本文质性地展示了所提出的模型能够学习到具有语义和句法意义的语言短语表示。研究背景方面，深度神经网络在多种应用领域中取得了巨大成功，例如图像识别和语音识别。此外，许多近期的研究显示神经网络可以成功应用于自然语言处理中的多项任务，包括语言建模、同义词检测以及词嵌入提取等。在统计机器翻译领域，深度神经网络的应用也展现出良好的前景。 #### RNN编码器-解码器架构 ##### 架构概述 RNN编码器-解码器模型是一种创新的神经网络架构，它通过两个RNN来实现序列到序列的学习。第一个RNN（编码器）接收输入序列并将其转换为一个固定长度的向量；第二个RNN（解码器）则根据这个向量生成输出序列。这种架构允许模型处理不同长度的输入和输出序列，并且可以有效地捕获序列中的依赖关系。 ##### 工作原理 - **编码过程**：输入序列被逐个元素送入编码器RNN，每个时间步的隐藏状态都取决于当前输入和前一时间步的隐藏状态。最后一步的隐藏状态被视为整个输入序列的紧凑表示。 - **解码过程**：解码器RNN利用编码器生成的向量来生成输出序列。初始状态下，解码器RNN的状态设置为编码器的最后一个隐藏状态。之后，解码器在每个时间步骤上生成输出序列的一个元素，并更新其内部状态。 ##### 训练过程模型的训练是通过最大化目标序列给定源序列的条件概率来完成的。这意味着模型被训练去预测下一个词的概率分布，基于已有的上下文信息。具体而言，训练时使用教师强制（teacher forcing）技术，即在每个时间步上向解码器提供真实的前一个词作为输入，而不是模型自己生成的词。这样可以减少训练过程中“错误积累”的问题。 #### 实验结果与分析 ##### 性能提升实验结果显示，在传统的短语基统计机器翻译系统中加入RNN编码器-解码器模型计算的短语对条件概率作为额外特征后，翻译质量有了显著提高。这意味着，即使是简单的添加一个新特征，也可以带来实质性的改进。 ##### 语义和句法表示进一步的分析表明，该模型能够学习到有意义的语义和句法表示。这体现在模型生成的短语对中，它们不仅在语义上与原始短语相似，而且在句法结构上也保持一致。这对于提高翻译质量和理解能力至关重要。 #### 结论与展望 RNN编码器-解码器模型为统计机器翻译领域带来了新的突破。通过学习短语的表示，模型能够更好地捕捉到语言中的语义和句法结构，从而提高翻译的准确性和流畅性。未来的研究可以探索更复杂的模型结构和训练策略，以进一步提高模型的表现。此外，结合其他NLP任务和技术（如注意力机制）也有望为机器翻译带来更多创新。

资源推荐

资源详情

资源评论

Learning Phrase Representations using RNN Encoder–Decoder

for Statistical Machine Translation

Kyunghyun Cho

Bart van Merri

enboer Caglar Gulcehre

Universit

e de Montr

eal

firstname.lastname@umontreal.ca

Dzmitry Bahdanau

Jacobs University, Germany

d.bahdanau@jacobs-university.de

Fethi Bougares Holger Schwenk

Universit

e du Maine, France

firstname.lastname@lium.univ-lemans.fr

Yoshua Bengio

Universit

e de Montr

eal, CIFAR Senior Fellow

find.me@on.the.web

Abstract

In this paper, we propose a novel neu-

ral network model called RNN Encoder–

Decoder that consists of two recurrent

neural networks (RNN). One RNN en-

codes a sequence of symbols into a ﬁxed-

length vector representation, and the other

decodes the representation into another se-

quence of symbols. The encoder and de-

coder of the proposed model are jointly

trained to maximize the conditional prob-

ability of a target sequence given a source

sequence. The performance of a statisti-

cal machine translation system is empiri-

cally found to improve by using the con-

ditional probabilities of phrase pairs com-

puted by the RNN Encoder–Decoder as an

additional feature in the existing log-linear

model. Qualitatively, we show that the

proposed model learns a semantically and

syntactically meaningful representation of

linguistic phrases.

1 Introduction

Deep neural networks have shown great success in

various applications such as objection recognition

(see, e.g., (Krizhevsky et al., 2012)) and speech

recognition (see, e.g., (Dahl et al., 2012)). Fur-

thermore, many recent works showed that neu-

ral networks can be successfully used in a num-

ber of tasks in natural language processing (NLP).

These include, but are not limited to, language

modeling (Bengio et al., 2003), paraphrase detec-

tion (Socher et al., 2011) and word embedding ex-

traction (Mikolov et al., 2013). In the ﬁeld of sta-

tistical machine translation (SMT), deep neural

networks have begun to show promising results.

(Schwenk, 2012) summarizes a successful usage

of feedforward neural networks in the framework

of phrase-based SMT system.

Along this line of research on using neural net-

works for SMT, this paper focuses on a novel neu-

ral network architecture that can be used as a part

of the conventional phrase-based SMT system.

The proposed neural network architecture, which

we will refer to as an RNN Encoder–Decoder, con-

sists of two recurrent neural networks (RNN) that

act as an encoder and a decoder pair. The en-

coder maps a variable-length source sequence to a

ﬁxed-length vector, and the decoder maps the vec-

tor representation back to a variable-length target

sequence. The two networks are trained jointly to

maximize the conditional probability of the target

sequence given a source sequence. Additionally,

we propose to use a rather sophisticated hidden

unit in order to improve both the memory capacity

and the ease of training.

The proposed RNN Encoder–Decoder with a

novel hidden unit is empirically evaluated on the

task of translating from English to French. We

train the model to learn the translation probabil-

ity of an English phrase to a corresponding French

phrase. The model is then used as a part of a stan-

dard phrase-based SMT system by scoring each

phrase pair in the phrase table. The empirical eval-

uation reveals that this approach of scoring phrase

pairs with an RNN Encoder–Decoder improves

the translation performance.

We qualitatively analyze the trained RNN

Encoder–Decoder by comparing its phrase scores

with those given by the existing translation model.

The qualitative analysis shows that the RNN

Encoder–Decoder is better at capturing the lin-

guistic regularities in the phrase table, indirectly

explaining the quantitative improvements in the

overall translation performance. The further anal-

ysis of the model reveals that the RNN Encoder–

Decoder learns a continuous space representation

of a phrase that preserves both the semantic and

syntactic structure of the phrase.

arXiv:1406.1078v3 [cs.CL] 3 Sep 2014

2 RNN Encoder–Decoder

2.1 Preliminary: Recurrent Neural Networks

A recurrent neural network (RNN) is a neural net-

work that consists of a hidden state h and an

optional output y which operates on a variable-

length sequence x = (x

, . . . , x

). At each time

step t, the hidden state h

hti

of the RNN is updated

hti

= f



ht−1i

, x



, (1)

where f is a non-linear activation func-

tion. f may be as simple as an element-

wise logistic sigmoid function and as com-

plex as a long short-term memory (LSTM)

unit (Hochreiter and Schmidhuber, 1997).

An RNN can learn a probability distribution

over a sequence by being trained to predict the

next symbol in a sequence. In that case, the output

at each timestep t is the conditional distribution

p(x

| x

t−1

, . . . , x

). For example, a multinomial

distribution (1-of-K coding) can be output using a

softmax activation function

p(x

t,j

= 1 | x

t−1

, . . . , x

) =

exp



hti



exp



hti



(2)

for all possible symbols j = 1, . . . , K, where w

are the rows of a weight matrix W. By combining

these probabilities, we can compute the probabil-

ity of the sequence x using

p(x) =

t=1

p(x

| x

t−1

, . . . , x

). (3)

From this learned distribution, it is straightfor-

ward to sample a new sequence by iteratively sam-

pling a symbol at each time step.

2.2 RNN Encoder–Decoder

In this paper, we propose a novel neural network

architecture that learns to encode a variable-length

sequence into a ﬁxed-length vector representation

and to decode a given ﬁxed-length vector rep-

resentation back into a variable-length sequence.

From a probabilistic perspective, this new model

is a general method to learn the conditional dis-

tribution over a variable-length sequence condi-

tioned on yet another variable-length sequence,

e.g. p(y

, . . . , y

| x

, . . . , x

), where one

Decoder

Encoder

Figure 1: An illustration of the proposed RNN

Encoder–Decoder.

should note that the input and output sequence

lengths T and T

may differ.

The encoder is an RNN that reads each symbol

of an input sequence x sequentially. As it reads

each symbol, the hidden state of the RNN changes

according to Eq. (1). After reading the end of

the sequence (marked by an end-of-sequence sym-

bol), the hidden state of the RNN is a summary c

of the whole input sequence.

The decoder of the proposed model is another

RNN which is trained to generate the output se-

quence by predicting the next symbol y

given the

hidden state h

hti

. However, unlike the RNN de-

scribed in Sec. 2.1, both y

and h

hti

are also con-

ditioned on y

t−1

and on the summary c of the input

sequence. Hence, the hidden state of the decoder

at time t is computed by,

hti

= f



ht−1i

, y

t−1

, c



and similarly, the conditional distribution of the

next symbol is

P (y

t−1

, y

t−2

, . . . , y

, c) = g



hti

, y

t−1

, c



for given activation functions f and g (the latter

must produce valid probabilities, e.g. with a soft-

max).

See Fig. 1 for a graphical depiction of the pro-

posed model architecture.

The two components of the proposed RNN

Encoder–Decoder are jointly trained to maximize

the conditional log-likelihood

max

n=1

log p

| x

), (4)

where θ is the set of the model parameters and

each (x

, y

) is an (input sequence, output se-

quence) pair from the training set. In our case,

as the output of the decoder, starting from the in-

put, is differentiable, we can use a gradient-based

algorithm to estimate the model parameters.

Once the RNN Encoder–Decoder is trained, the

model can be used in two ways. One way is to use

the model to generate a target sequence given an

input sequence. On the other hand, the model can

be used to score a given pair of input and output

sequences, where the score is simply a probability

(y | x) from Eqs. (3) and (4).

2.3 Hidden Unit that Adaptively Remembers

and Forgets

In addition to a novel model architecture, we also

propose a new type of hidden unit (f in Eq. (1))

that has been motivated by the LSTM unit but is

much simpler to compute and implement.

Fig. 2

shows the graphical depiction of the proposed hid-

den unit.

Let us describe how the activation of the j-th

hidden unit is computed. First, the reset gate r

computed by

= σ





ht−1i





, (5)

where σ is the logistic sigmoid function, and [.]

denotes the j-th element of a vector. x and h

t−1

are the input and the previous hidden state, respec-

tively. W

and U

are weight matrices which are

learned.

Similarly, the update gate z

is computed by

= σ





ht−1i





. (6)

The actual activation of the proposed unit h

then computed by

hti

= z

ht−1i

+ (1 − z

)

hti

, (7)

where

hti

= φ



[Wx]





r  h

ht−1i





. (8)

In this formulation, when the reset gate is close

to 0, the hidden state is forced to ignore the pre-

vious hidden state and reset with the current input

The LSTM unit, which has shown impressive results in

several applications such as speech recognition, has a mem-

ory cell and four gating units that adaptively control the in-

formation ﬂow inside the unit, compared to only two gating

units in the proposed hidden unit. For details on LSTM net-

works, see, e.g., (Graves, 2012).

Figure 2: An illustration of the proposed hidden

activation function. The update gate z selects

whether the hidden state is to be updated with

a new hidden state

h. The reset gate r decides

whether the previous hidden state is ignored. See

Eqs. (5)–(8) for the detailed equations of r, z, h

and

only. This effectively allows the hidden state to

drop any information that is found to be irrelevant

later in the future, thus, allowing a more compact

representation.

On the other hand, the update gate controls how

much information from the previous hidden state

will carry over to the current hidden state. This

acts similarly to the memory cell in the LSTM

network and helps the RNN to remember long-

term information. Furthermore, this may be con-

sidered an adaptive variant of a leaky-integration

unit (Bengio et al., 2013).

As each hidden unit has separate reset and up-

date gates, each hidden unit will learn to capture

dependencies over different time scales. Those

units that learn to capture short-term dependencies

will tend to have reset gates that are frequently ac-

tive, but those that capture longer-term dependen-

cies will have update gates that are mostly active.

In our preliminary experiments, we found that

it is crucial to use this new unit with gating units.

We were not able to get meaningful result with an

oft-used tanh unit without any gating.

3 Statistical Machine Translation

In a commonly used statistical machine translation

system (SMT), the goal of the system (decoder,

speciﬁcally) is to ﬁnd a translation f given a source

sentence e, which maximizes

p(f | e) ∝ p(e | f)p(f ),

where the ﬁrst term at the right hand side is called

translation model and the latter language model

(see, e.g., (Koehn, 2005)). In practice, however,

most SMT systems model log p(f | e) as a log-

linear model with additional features and corre-

剩余14页未读，继续阅读

评论收藏

内容反馈

xiewenbo

粉丝: 385
资源: 6

Learning Phrase Representations using RNN Encoder–Decoder

Learning representations by back propagating errors(1985)

基于循环神经网络RNN的歌词生成

Learning-phase

Encoder-Decoder①

Seq2Seq案例源代码

自然语言处理.zip

Guess My Phrase Game using JavaScript with Source Code.zip

wpa_passphrase.rar_tool_wpa_passphrase

Representation Learning for Word, Sense, Phrase, Document and Knowledge-刘知远

Android代码-phrase

融合位置特征的关键短语集合抽取模型.pdf

Feature Engineering for Machine Learning

matlab开发-phrase2wordm

wpa_supplicant，wpa_passphrase源码

Custom-phrase.txt

english phrase

Java Phrase Book

Laravel开发-phrase-app-export

atom-phrase:收集短语并快速参考

Efficient Phrase Querying with an Auxiliary Index

phrase机制介绍.PNG

论文研究-基于有限域上Chebyshev多项式的密钥协商方案.pdf

PyPI 官网下载 | django-phrase-1.0.2.tar.gz

PyPI 官网下载 | django-phrase-0.0.2.tar.gz

s19_phrase.zip

HTML4.rar_php_phrase6z8

PyPI 官网下载 | phrase_pickers-0.2.8.tar.gz

优课UOOC 深圳大学 大学英语3 - 第2章 A quiz on the phrase put on 答案

passphrase:使用 Diceware 方法生成密码短语的 Ruby 库和命令行工具

最新资源

优课UOOC 深圳大学大学英语3 - 第2章 A quiz on the phrase put on 答案