基于Spark的音乐数据分析系统论文.docx

音乐数据分析系统的设计与实现

摘要

随着城市人口规模的迅速增长，我国的城市轨道交通的压力也日益增加，并且随着

城市经济的迅速发展，人们对于音乐娱乐出行的依赖程度越来越高。音乐作为音乐娱乐的

主要工具，各个线路、站点的数据量规模不断攀升且呈现出较大差异。随之而来也产生了

海量的音乐刷卡数据，这些数据对于构建高效、安全、便捷的交通出行有着至关重要的作

用。机器学习领域中的算法可以从这些历史数据中预测出数据分布特点，快速和准确的对

音乐短期数据进行预测。我们不仅需要了解音乐数据线路、站点的分布情况，更需要进一

步的对线路各站点的数据量进行精确的预测。

MySQL 数据库；4.利用 Spark MLlib 进行数据和关系预测；5.利用 IntelliJ IDEA 搭建动态 Web

应用；6.利用 plotly 进行前端可视化分析。

关键词：音乐数据分析；可视化分析；python 语言

Abstract

With the rapid growth of urban population, the pressure of urban rail transit in China is also

increasing, and with the rapid development of urban economy, people rely more and more on

urban transportation. As the main means of urban transportation, the passenger flow scale of

each line and station is increasing and showing great differences. Thereafter, a huge amount of

subway card swiping data has been generated, which plays a vital role in building efficient, safe

and convenient transportation. Algorithms in the field of machine learning can predict the

subway, focuses on the short-term passenger flow prediction problem in Hangzhou subway,

studies and designs the mechanism for feature screening and fusion to build effective features,

and on this basis proposes the algorithm model of short-term passenger flow prediction to

achieve accurate short-term passenger flow prediction, The prediction results are applied and

analyzed.

This paper mainly analyzes the data of Shenzhen Metro. The system technology is mainly

used: 1. Pre-processing the original data set; 2. Load subway traffic data into Hadoop cluster

HDFS; 3. Write Spark program in python language to process and analyze the data in HDFS, and

write the analysis results into MySQL database; 4. Use Spark MLlib to forecast transaction

behavior and relationship; 5. Use IntelliJ IDEA to build dynamic web applications; 6. Use plotly

chart display, subway wandering restriction pie chart analysis of the top 10 stations, and subway

current restriction histogram analysis of 10 time periods.

Key words: subway data analysis; Visual analysis; Python language

目录

1 绪论 .....................................................................................................................................1

1.1 研究背景 ..................................................................................................................1

1.2 研究意义 ..................................................................................................................2

1.3 研究现状 ..................................................................................................................3

1.3.1 国内外研究现状分析 ...................................................................................3

1.3.2 国外研究现状 ...............................................................................................3

2.1HDFS 集群..................................................................................................................5

2.2python........................................................................................................................5

2.3spark ..........................................................................................................................5

3.3 用例分析 ..................................................................................................................8

4 系统数据库 .......................................................................................................................10

4.1 数据库表单 ............................................................................................................10

4.2 系统 E-R 图.............................................................................................................13

4 系统详细设计 ...................................................................................................................15

4.1 系统架构 ................................................................................................................15

4.2 爬虫分析 ................................................................................................................15

4.3 数据可视化 ............................................................................................................16

4.4 系统登录流程 ........................................................................................................17

4.5 预测分析流程 ........................................................................................................18

5.系统实现 ...........................................................................................................................19

6 系统测试 ..........................................................................................................................23

6.1 测试目的 .............................................................................................................23

6.2 测试方法 .............................................................................................................23

6.3 登录测试 .............................................................................................................23

总结 ......................................................................................................................................25

参考文献 ..............................................................................................................................26

致 谢 ..................................................................................................................................28