arXiv:1607.01759v3 [cs.CL] 9 Aug 2016
Bag of Tricks for Efficient Text Classification
Armand Joulin Edouard Grave Piotr Bojanowski Tomas Mikolov
Facebook AI Research
{ajoulin,egrave,bojanowski,tmikolov}@fb.com
Abstract
This paper explores a simple and efficient
baseline for text classification. Our ex-
periments show that our fast text classi-
fier fastText is often on par with deep
learning classifiers in terms of accuracy, and
many orders of magnitude faster for training
and evaluation. We can train fastText on
more than one billion words in less than ten
minutes using a standard multicore CPU, and
classify half a million sentences among 312K
classes in less than a minute.
1 Introduction
Text classification is an important task in Natural
Language Processing with many applications, such
as web search, information retrieval, ranking and
document classification (Deerwester et al., 1990;
Pang and Lee, 2008). Recently, models based
on neural networks have become increasingly
popular (Kim, 2014; Zhang and LeCun, 2015;
Conneau et al., 2016). While these models achieve
very good performance in practice, they tend to be
relatively slow both at train and test time, limiting
their use on very large datasets.
Meanwhile, linear classifiers are of-
ten considered as strong baselines for text
classification problems (Joachims, 1998;
McCallum and Nigam, 1998; Fan et al., 2008).
Despite their simplicity, they often obtain state-
of-the-art performances if the right features are
used (Wang and Manning, 2012). They also
have the potential to scale to very large cor-
pus (Agarwal et al., 2014).
In this work, we explore ways to scale these
baselines to very large corpus with a large output
space, in the context of text classification. Inspired
by the recent work in efficient word representation
learning (Mikolov et al., 2013; Levy et al., 2015),
we show that linear models with a rank constraint
and a fast loss approximation can train on a billion
words within ten minutes, while achieving perfor-
mance on par with the state-of-the-art. We evalu-
ate the quality of our approach fastText
1
on two
different tasks, namely tag prediction and sentiment
analysis.
2 Model architecture
A simple and efficient baseline for sentence
classification is to represent sentences as bag of
words (BoW) and train a linear classifier, e.g., a
logistic regression or an SVM (Joachims, 1998;
Fan et al., 2008). However, linear classifiers do
not share parameters among features and classes.
This possibly limits their generalization in the
context of large output space where some classes
have very few examples. Common solutions
to this problem are to factorize the linear clas-
sifier into low rank matrices (Schutze, 1992;
Mikolov et al., 2013) or to use multilayer
neural networks (Collobert and Weston, 2008;
Zhang et al., 2015).
Figure 1 shows a simple linear model with rank
constraint. The first weight matrix A is a look-up
table over the words. The word representations are
then averaged into a text representation, which is in
turn fed to a linear classifier. The text representa-
1
https://github.com/facebookresearch/fastText
资源评论
java的世界
- 粉丝: 0
- 资源: 6
最新资源
- 基于simulink的OFDM-BPSK通信系统误码率仿真，包括simulink模型，仿真操作步骤视频
- MM 收发货备份程序ZZZZZZZZZ
- LVGL-lib(LVGL库文件)
- 【python毕业设计】基于RSA加密算法软件的研究设计源码（完整前后端+mysql+说明文档+LW）.zip
- 雷赛驱动器参数设置2024 LD5-ACHSeries 伺服驱动器调试软件设置原点及定位值：
- 【python毕业设计】基于des算法的企业用户数据安全软件源码（完整前后端+mysql+说明文档+LW）.zip
- Linux自动化运维课程中Awk编程语言的应用实践与解析
- 【python毕业设计】基于B站用户行为分析系统源码（完整前后端+mysql+说明文档+LW）.zip
- RMc板和达妙板通信的英雄代码
- 【python毕业设计】反爬虫技术的研究源码（完整前后端+mysql+说明文档+LW）.zip
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
