成都理工大学硕士学位论文
II
Research on Real-time Movie Recommender System Based on
Spark
Abstract
With the advent of the Internet era, Web2.0 and the Internet of things technology
are developing rapidly, and more and more data are generated every year in the world.
How to help people find information that they are interested in from these massive
data, and bring more profits for the merchants and achieve the win-win situation of
consumers and businesses, is of great practical significance. In order to meet this
widespread demand, there are two technologies to solve this problem in the Internet
industry, one is the well-known search engine technology, and the other is our daily
contact recommendation system technology. Although on the Internet, the application
of recommendation system is very wide, there are many algorithms and mature
theories, but there are still many problems and challenges, such as: data sparsity,
recommendation quality and recommendation real-time, cold start. In order to solve
these problems effectively, this paper discusses the two aspects of recommendation
quality and recommendation real-time. First, it introduces the relevant background of
the theory, and then discusses the related large data processing techniques, including
the major data technologies, such as distributed file system (HDFS), memory
computing engine (Spark) and message queue (Kafka), and also introduces the related
algorithms in the recommendation system, and explores the application of deep
learning technology. In the recommendation system, the strong fitting ability of the
neural network is used to improve the quality of the recommendation. Finally, a
simple real-time movie recommendation system is realized by using the popular data
processing technology. The main achievements of this article are as follows:
(1) In the recommendation quality problem, the implicit feedback data Lai
Jianmo of the user is fully utilized, and the depth learning related technology is used
in the recommendation algorithm, and a neural network structure is used to replace
the inner product of the eigenvector of the hidden space. The collaborative filtering
algorithm is formalized by the neural network model and indirectly from the user's
implicit formula. The user's preferences are deduced from the feedback, thus
improving the quality of recommendation. Experiments were carried out on the
MovieLens dataset to verify the validity of the model, and the influence of hidden
评论0