推荐系统论文集合_xgboost精排推荐算法资源-CSDN文库

共10个文件

pdf：10个

机器学习

GBDT

deep

需积分: 29 127 浏览量 2019-02-19 11:23:29 上传评论 2 收藏 4.79MB ZIP 举报

资源推荐

资源详情

资源评论

收起资源包目录

推荐系统算法论文集.zip （10个子文件）

8. xgboost.pdf 923KB

1. BPR algorithm.pdf 294KB

3. deep&cross learning for ad click prediction.pdf 381KB

7. item-based collaborative filtering.pdf 264KB

10. DeepFM.pdf 1.14MB

2. gbdt+lr.pdf 774KB

4. wide and deep network.pdf 400KB

5. factorization machine.pdf 187KB

9. Deep Neural Networks for YouTube Recommendations.pdf 877KB

6. field-aware factorization machine.pdf 361KB

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Huifeng Guo

∗1

, Ruiming Tang

, Yunming Ye

†1

, Zhenguo Li

, Xiuqiang He

Shenzhen Graduate School, Harbin Institute of Technology, China

Noah’s Ark Research Lab, Huawei, China

huifengguo@yeah.net, yeyunming@hit.edu.cn

{tangruiming, li.zhenguo, hexiuqiang}@huawei.com

Abstract

Learning sophisticated feature interactions behind

user behaviors is critical in maximizing CTR for

recommender systems. Despite great progress, ex-

isting methods seem to have a strong bias towards

low- or high-order interactions, or require exper-

tise feature engineering. In this paper, we show

that it is possible to derive an end-to-end learn-

ing model that emphasizes both low- and high-

order feature interactions. The proposed model,

DeepFM, combines the power of factorization ma-

chines for recommendation and deep learning for

feature learning in a new neural network architec-

ture. Compared to the latest Wide & Deep model

from Google, DeepFM has a shared input to its

“wide” and “deep” parts, with no need of feature

engineering besides raw features. Comprehensive

experiments are conducted to demonstrate the ef-

fectiveness and efﬁciency of DeepFM over the ex-

isting models for CTR prediction, on both bench-

mark data and commercial data.

1 Introduction

The prediction of click-through rate (CTR) is critical in rec-

ommender system, where the task is to estimate the probabil-

ity a user will click on a recommended item. In many recom-

mender systems the goal is to maximize the number of clicks,

and so the items returned to a user can be ranked by estimated

CTR; while in other application scenarios such as online ad-

vertising it is also important to improve revenue, and so the

ranking strategy can be adjusted as CTR×bid across all can-

didates, where “bid” is the beneﬁt the system receives if the

item is clicked by a user. In either case, it is clear that the key

is in estimating CTR correctly.

It is important for CTR prediction to learn implicit feature

interactions behind user click behaviors. By our study in a

mainstream apps market, we found that people often down-

load apps for food delivery at meal-time, suggesting that the

(order-2) interaction between app category and time-stamp

∗

This work is done when Huifeng Guo worked as intern at

Noah’s Ark Research Lab, Huawei.

†

Corresponding Author.

Figure 1: Wide & deep architecture of DeepFM. The wide and deep

component share the same input raw feature vector, which enables

DeepFM to learn low- and high-order feature interactions simulta-

neously from the input raw features.

can be used as a signal for CTR. As a second observation,

male teenagers like shooting games and RPG games, which

means that the (order-3) interaction of app category, user gen-

der and age is another signal for CTR. In general, such inter-

actions of features behind user click behaviors can be highly

sophisticated, where both low- and high-order feature interac-

tions should play important roles. According to the insights

of the Wide & Deep model

[

Cheng et al., 2016

]

from google,

considering low- and high-order feature interactions simulta-

neously brings additional improvement over the cases of con-

sidering either alone.

The key challenge is in effectively modeling feature inter-

actions. Some feature interactions can be easily understood,

thus can be designed by experts (like the instances above).

However, most other feature interactions are hidden in data

and difﬁcult to identify a priori (for instance, the classic as-

sociation rule “diaper and beer” is mined from data, instead

of discovering by experts), which can only be captured auto-

matically by machine learning. Even for easy-to-understand

interactions, it seems unlikely for experts to model them ex-

haustively, especially when the number of features is large.

Despite their simplicity, generalized linear models, such as

FTRL

[

McMahan et al., 2013

]

, have shown decent perfor-

mance in practice. However, a linear model lacks the abil-

ity to learn feature interactions, and a common practice is

to manually include pairwise feature interactions in its fea-

ture vector. Such a method is hard to generalize to model

high-order feature interactions or those never or rarely appear

in the training data

[

Rendle, 2010

]

. Factorization Machines

arXiv:1703.04247v1 [cs.IR] 13 Mar 2017

(FM)

[

Rendle, 2010

]

model pairwise feature interactions as

inner product of latent vectors between features and show

very promising results. While in principle FM can model

high-order feature interaction, in practice usually only order-

2 feature interactions are considered due to high complexity.

As a powerful approach to learning feature representa-

tion, deep neural networks have the potential to learn so-

phisticated feature interactions. Some ideas extend CNN

and RNN for CTR predition

[

Liu et al., 2015; Zhang et

al., 2014

]

, but CNN-based models are biased to the in-

teractions between neighboring features while RNN-based

models are more suitable for click data with sequential de-

pendency.

[

Zhang et al., 2016

]

studies feature representa-

tions and proposes Factorization-machine supported Neural

Network (FNN). This model pre-trains FM before applying

DNN, thus limited by the capability of FM. Feature interac-

tion is studied in

[

Qu et al., 2016

]

, by introducing a prod-

uct layer between embedding layer and fully-connected layer,

and proposing the Product-based Neural Network (PNN). As

noted in

[

Cheng et al., 2016

]

, PNN and FNN, like other deep

models, capture little low-order feature interactions, which

are also essential for CTR prediction. To model both low-

and high-order feature interactions,

[

Cheng et al., 2016

]

pro-

poses an interesting hybrid network structure (Wide & Deep )

that combines a linear (“wide”) model and a deep model. In

this model, two different inputs are required for the “wide

part” and “deep part”, respectively, and the input of “wide

part” still relies on expertise feature engineering.

One can see that existing models are biased to low- or high-

order feature interaction, or rely on feature engineering. In

this paper, we show it is possible to derive a learning model

that is able to learn feature interactions of all orders in an end-

to-end manner, without any feature engineering besides raw

features. Our main contributions are summarized as follows:

• We propose a new neural network model DeepFM

(Figure 1) that integrates the architectures of FM and

deep neural networks (DNN). It models low-order fea-

ture interactions like FM and models high-order fea-

ture interactions like DNN. Unlike the wide & deep

model

[

Cheng et al., 2016

]

, DeepFM can be trained end-

to-end without any feature engineering.

• DeepFM can be trained efﬁciently because its wide part

and deep part, unlike

[

Cheng et al., 2016

]

, share the

same input and also the embedding vector. In

[

Cheng et

al., 2016

]

, the input vector can be of huge size as it in-

cludes manually designed pairwise feature interactions

in the input vector of its wide part, which also greatly

increases its complexity.

• We evaluate DeepFM on both benchmark data and com-

mercial data, which shows consistent improvement over

existing models for CTR prediction.

2 Our Approach

Suppose the data set for training consists of n instances

(χ, y), where χ is an m-ﬁelds data record usually involving

a pair of user and item, and y ∈ {0, 1} is the associated la-

bel indicating user click behaviors (y = 1 means the user

clicked the item, and y = 0 otherwise). χ may include cat-

egorical ﬁelds (e.g., gender, location) and continuous ﬁelds

(e.g., age). Each categorical ﬁeld is represented as a vec-

tor of one-hot encoding, and each continuous ﬁeld is repre-

sented as the value itself, or a vector of one-hot encoding af-

ter discretization. Then, each instance is converted to (x, y)

where x = [x

field

, x

field

, ..., x

filed

, ..., x

field

] is a d-

dimensional vector, with x

field

being the vector representa-

tion of the j-th ﬁeld of χ. Normally, x is high-dimensional

and extremely sparse. The task of CTR prediction is to build a

prediction model ˆy = CT R model(x) to estimate the prob-

ability of a user clicking a speciﬁc app in a given context.

2.1 DeepFM

We aim to learn both low- and high-order feature interactions.

To this end, we propose a Factorization-Machine based neu-

ral network (DeepFM). As depicted in Figure 1

, DeepFM

consists of two components, FM component and deep com-

ponent, that share the same input. For feature i, a scalar w

is used to weigh its order-1 importance, a latent vector V

used to measure its impact of interactions with other features.

is fed in FM component to model order-2 feature interac-

tions, and fed in deep component to model high-order feature

interactions. All parameters, including w

, V

, and the net-

work parameters (W

(l)

, b

(l)

below) are trained jointly for the

combined prediction model:

ˆy = sigmoid(y

F M

+ y

DN N

), (1)

where ˆy ∈ (0, 1) is the predicted CTR, y

F M

is the output of

FM component, and y

DN N

is the output of deep component.

FM Component

Figure 2: The architecture of FM.

The FM component is a factorization machine, which

is proposed in

[

Rendle, 2010

]

to learn feature interactions

for recommendation. Besides a linear (order-1) interactions

among features, FM models pairwise (order-2) feature inter-

actions as inner product of respective feature latent vectors.

In all Figures of this paper, a Normal Connection in black

refers to a connection with weight to be learned; a Weight-1 Con-

nection, red arrow, is a connection with weight 1 by default; Em-

bedding, blue dashed arrow, means a latent vector to be learned; Ad-

dition means adding all input together; Product, including Inner-

and Outer-Product, means the output of this unit is the product of

two input vector; Sigmoid Function is used as the output function

in CTR prediction; Activation Functions, such as relu and tanh, are

used for non-linearly transforming the signal.

It can capture order-2 feature interactions much more effec-

tively than previous approaches especially when the dataset is

sparse. In previous approaches, the parameter of an interac-

tion of features i and j can be trained only when feature i and

feature j both appear in the same data record. While in FM, it

is measured via the inner product of their latent vectors V

and

. Thanks to this ﬂexible design, FM can train latent vector

) whenever i (or j) appears in a data record. Therefore,

feature interactions, which are never or rarely appeared in the

training data, are better learnt by FM.

As Figure 2 shows, the output of FM is the summation of

an Addition unit and a number of Inner Product units:

F M

= hw, xi +

, V

i x

· x

, (2)

where w ∈ R

and V

∈ R

(k is given)

. The Addition

unit (hw, xi) reﬂects the importance of order-1 features, and

the Inner Product units represent the impact of order-2 feature

interactions.

Deep Component

Figure 3: The architecture of DNN.

The deep component is a feed-forward neural network,

which is used to learn high-order feature interactions. As

shown in Figure 3, a data record (a vector) is fed into the neu-

ral network. Compared to neural networks with image

[

et al., 2016

]

or audio

[

Boulanger-Lewandowski et al., 2013

]

data as input, which is purely continuous and dense, the in-

put of CTR prediction is quite different, which requires a

new network architecture design. Speciﬁcally, the raw fea-

ture input vector for CTR prediction is usually highly sparse

super high-dimensional

, categorical-continuous-mixed, and

grouped in ﬁelds (e.g., gender, location, age). This suggests

an embedding layer to compress the input vector to a low-

dimensional, dense real-value vector before further feeding

into the ﬁrst hidden layer, otherwise the network can be over-

whelming to train.

Figure 4 highlights the sub-network structure from the in-

put layer to the embedding layer. We would like to point out

the two interesting features of this network structure: 1) while

the lengths of different input ﬁeld vectors can be different,

We omit a constant offset for simplicity.

Only one entry is non-zero for each ﬁeld vector.

E.g., in an app store of billion users, the one ﬁeld vector for user

ID is already of billion dimensions.

Figure 4: The structure of the embedding layer

their embeddings are of the same size (k); 2) the latent fea-

ture vectors (V ) in FM now server as network weights which

are learned and used to compress the input ﬁeld vectors to the

embedding vectors. In

[

Zhang et al., 2016

]

, V is pre-trained

by FM and used as initialization. In this work, rather than us-

ing the latent feature vectors of FM to initialize the networks

as in

[

Zhang et al., 2016

]

, we include the FM model as part of

our overall learning architecture, in addition to the other DNN

model. As such, we eliminate the need of pre-training by FM

and instead jointly train the overall network in an end-to-end

manner. Denote the output of the embedding layer as:

(0)

= [e

, e

, ..., e

], (3)

where e

is the embedding of i-th ﬁeld and m is the number

of ﬁelds. Then, a

(0)

is fed into the deep neural network, and

the forward process is:

(l+1)

= σ(W

(l)

+ b

(l)

), (4)

where l is the layer depth and σ is an activation function. a

(l)

, b

(l)

are the output, model weight, and bias of the l-th

layer. After that, a dense real-value feature vector is gener-

ated, which is ﬁnally fed into the sigmoid function for CTR

prediction: y

DN N

= σ(W

|H|+1

· a

+ b

|H|+1

), where |H|

is the number of hidden layers.

It is worth pointing out that FM component and deep com-

ponent share the same feature embedding, which brings two

important beneﬁts: 1) it learns both low- and high-order fea-

ture interactions from raw features; 2) there is no need for ex-

pertise feature engineering of the input, as required in Wide

& Deep

[

Cheng et al., 2016

]

2.2 Relationship with the other Neural Networks

Inspired by the enormous success of deep learning in var-

ious applications, several deep models for CTR prediction

are developed recently. This section compares the proposed

DeepFM with existing deep models for CTR prediction.

FNN: As Figure 5 (left) shows, FNN is a FM-initialized feed-

forward neural network

[

Zhang et al., 2016

]

. The FM pre-

training strategy results in two limitations: 1) the embedding

parameters might be over affected by FM; 2) the efﬁciency is

reduced by the overhead introduced by the pre-training stage.

In addition, FNN captures only high-order feature interac-

tions. In contrast, DeepFM needs no pre-training and learns

both high- and low-order feature interactions.

PNN: For the purpose of capturing high-order feature interac-

tions, PNN imposes a product layer between the embedding

layer and the ﬁrst hidden layer

[

Qu et al., 2016

]

. According

to different types of product operation, there are three vari-

ants: IPNN, OPNN, and PNN∗, where IPNN is based on in-

ner product of vectors, OPNN is based on outer product, and

PNN∗ is based on both inner and outer products.

评论收藏

内容反馈

dreamfantacy

粉丝: 6
资源: 5

推荐系统论文集合

推荐系统最全论文集合

推荐系统论文

recsys 推荐系统大会 2016 论文集 共105篇

WWW 2020 推荐系统 相关的论文.zip

5篇WWW 2020热门论文【推荐系统专题】.zip

WWW 2020热点论文（6篇）【推荐系统相关】.zip

融合协同过滤和XGBoost的推荐算法

推荐算法论文

嵌入式系统论文大集合

最新信息系统项目管理师模板论文集合

论文研究-集合竞价算法对股票价格的影响.pdf

系统分析师考试论文范文资料集合.rar

Recommended-system:推荐系统学习笔记，包括论文，代码实现，相关推荐比赛等

Recsys2018 总结 （推荐系统最新技术、应用和方向）32篇论文解读

论文查重网站集合

论文研究-基于双射软集合决策系统的参数约减.pdf

2017-2018年机器学习顶会优质论文集合

基于MATLAB图像处理的汽车牌照识别系统论文集合

系统分析师论文范文集

人工智能 AI 论文 集合

论文研究-描述离散动态系统的集合标识Petri网:STPNs.pdf

基于J2EE框架的论文大集合包含15个论文

自然语言处理论文集合

相机标定论文集合

信息系统项目管理师优秀论文80篇（薛大龙博士）.pdf

软件类专业实习报告+毕业论文等集合

三维渲染、开发论文集合

最新资源

recsys 推荐系统大会 2016 论文集共105篇

WWW 2020 推荐系统相关的论文.zip

Recsys2018 总结（推荐系统最新技术、应用和方向）32篇论文解读

人工智能 AI 论文集合