B 海量多语言迁移学习1

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164

Florence, Italy, July 28 - August 2, 2019.

c

2019 Association for Computational Linguistics

151

Massively Multilingual Transfer for NER

Afshin Rahimi

∗

Yuan Li

∗

guage processing, with most tasks requiring large

quantities of annotated corpora. The majority of

the world’s 6,000+ languages however have lim-

ited or no annotated text, and therefore much

of the progress in NLP has yet to be realised

widely. Cross-lingual transfer learning is a tech-

nique which can compensate for the dearth of

data, by transferring knowledge from high- to low-

resource languages, which has typically taken the

form of annotation projection over parallel corpora

Most methods proposed for cross-lingual trans-

fer rely on a single source language, which lim-

its the transferable knowledge to only one source.

Both authors contributed equally to this work.

The code and the datasets will be made available at

https://github.com/afshinrahimi/mmner.

source languages, on the grounds of the script,

word order, loan words etc, and transfer would

¨

¨

and Cohn, 2016; Plank and Agi

ever, to the best of our knowledge, none of these

approaches adequately account for the quality of

transfer, but rather “weight” the contribution of

each language uniformly.

zero-shot multilingual transfer, inspired by re-

search in truth inference in crowd-sourcing, a re-

adapts these ideas to a multilingual transfer set-

ting, whereby we learn the quality of transfer, and

language-specific transfer errors, in order to infer

the best labelling in the target language, as part of

a Bayesian graphical model. The key insight is

that while the majority of poor models make lots

of mistakes, these mistakes are diverse, while the

few good models consistently provide reliable in-

put. This allows the model to infer which are the

reliable models in an unsupervised manner, i.e.,

use this data: 1) estimate reliability parameters of

152

the Bayesian model, and 2) explicit model selec-

tion and fine-tuning of a low-resource supervised

model, thus allowing for more accurate modelling

of language specific parameters, such as charac-

ter embeddings, shown to be important in previous

work (Xie et al., 2018).

Experimenting on two NER corpora, one with

model transfer has highly variable performance,

and uniform ensembling often substantially under-

as follows. We assume a collection of H mod-

els, all trained in a high resource setting, denoted

els are not well matched to our target data setting,

for instance these may be trained on data from dif-

ferent domains, or on different languages, as we

evaluate in our experiments, where we use cross-

lingual embeddings for model transfer. This is a

grounds of practicality, or the similarity between

the model’s native data condition and the target,

and this model is used to label the target data; or

b) allowing all models to ‘vote’ in an classifier en-

semble, such that the most frequent outcome is

selected as the ensemble output. Unfortunately

neither of these approaches are very accurate in

where we show a fixed source language model

we propose the BEA

uns

method inspired by work

We limit our attention to transfer in a ‘black-box’ setting,

that is, given predictive models, but not assuming access to

their data, nor their implementation. This is the most flexible

scenario, as it allows for application to settings with closed

APIs, and private datasets. It does, however, preclude multi-

task learning, as the source models are assumed to be static.

153

plaining the observed predictions Y in the most

efficient way. Where several transfer models have

identical predictions, k, on an instance, this can be

explained by letting z

and the confusion

matrices of those transfer models assigning high

(2)

Although there is no explicit breaking of the symmetry

of the model, we initialise inference using the majority vote,

which results in a bias towards this solution.

O O B-ORG I-ORG O O ORG

O B-PER I-PER I-PER O PER O

O B-PER I-PER I-PER O PER O

logarithmic derivative of the gamma function. The

sets of rules (1) and (2) are applied alternately, to

update the values of E

, and

gence, when the difference in the ELBO between

uns

2012) to filter out the worst of the transfer models.

We adopt a simple strategy that first estimates the

confusion matrices for all transfer models on all

labels, then ranks them based on their mean recall

on different entity categories (elements on the di-

agonals of their confusion matrices), and then runs

on an assumption that the true annotations are in-

dependent from each other, which simplifies the

model but may generate undesired results. That

is, entities predicted by different transfer models

could be mixed, resulting in labels inconsistent

with the BIO scheme. Table 1 shows an exam-

among which at most one is correct as they over-

lap. However, the aggregated result in the token