毕业设计论文Hadoop+爬虫+Spark外卖配送分析及可视化系统.docx

摘要

随着社会的不断发展，互联网数据时代的到来，数据的背后是什么，数据有什么用，

怎么用庞大的数据来呈现出数据的价值，让我们一起去揭开它神秘的面纱。

本次以外卖配送分析及可视化系统爬取为例，介绍网络爬虫的基本原理，Python 环

境的搭建，PyCharm scrapy 模块的爬虫数据的运用，把获取到的数据进行清洗、整合，

储存数据到 MySQL，然后进行数据可视化的呈现，简单对呈现的图进行数据分析。

关 键 词：Python；MySQL；PyCharm；爬虫；数据分析；数据呈现。

Python environment, the use of crawler data in the PyCharm sketch module, cleaning and

integrating the acquired data, storing the data to MySQL, and then presenting data

visualization to simply analyze the data presented.

Keywords: Python; MySQL； PyCharm； Reptiles; Data analysis; Data presentation.

目录

摘要 .....................................................................I

Abstrac .................................................................II

1 绪论.........................................................................................................................................1

1.1 背景及意义 ......................................................................................................................2

1.1.2 目标及意义................................................................................................................3

1.4.2 技术路线....................................................................................................................6

1.4.3 实现途径.....................................................................................................................7

2 开发环境及技术......................................................................................................................8

2.1 软件硬件设备 ..................................................................................................................8

2.1.1 其他............................................................................................................................9

2.2 开发环境与工具 ............................................................................................................10

2.2.1 Python 简介 ..............................................................................................................10

2.2.2 爬虫简介..................................................................................................................11

2.2.3 Django 框架简介.....................................................................................................12

2.2.4 MySqL 数据库 .........................................................................................................13

3.2.1 数据获取方法..........................................................................................................19

3.2.2 数据预处理设计......................................................................................................20

5 系统设计...............................................................................................................................22

4.2 系统数据库设计.................................................................................................................21

4.2.1 数据库 E-R 图 ..........................................................................................................21

4.3 数据库表结构..............................................................................................................21

5 系统实现与结论...................................................................................................................22

5.2 管理员功能的实现 .........................................................................................................23