# DataBand数据帮 轻量级一站式大数据分析平台
项目启动于2020-10-26,持续更新中。
## 完整开发使用文档
[详情开发使用介绍](https://www.yuque.com/docs/share/162f64f8-bc27-41c1-818c-57f6a953fab1)
## 介绍
DataBand(数据帮),快速采集清洗,任务管理,实时流和批处理数据分析,数据可视化展现,快速数据模板开发,ETL工具集、数据科学等。是轻量级的一站式的大数据平台。
我们致力于通过提供智能应用程序、数据分析和咨询服务来提供最优解决方案。
### 软件架构
![架构](https://gitee.com/475660/databand/raw/master/files/DataBand_structure.png "DataBand数据帮系统架构")
### 技术栈
#### 存储
- 分布式存储:HDFS、HBase
- 行式关系存储:MySQL、Oracle
- 列式存储:ClickHouse
- 列族存储:HBase、Cassandra
- 文档库:ElasticSearch、MongoDB
#### 计算
- 计算引擎:Presto、Hive
- 流处理:Storm、Flink
#### 集成:
- Flume
- Filebeat
- Logstash
#### 前端技术栈
- Vue
- Element UI
#### 后端技术栈
- Spring Boot
- Spring Cloud
- MyBatis
### 工程说明
#### 大数据模拟数据源生成数据(数据准备工程)
数据源:
![数据源](https://gitee.com/475660/databand/raw/master/files/DataBanddatasource.png "DataBand数据源")
- databand-mock-api:接口模拟工具,模拟业务系统api;
- databand-mock-log:日志模拟工具,手动产生大量的日志数据供调试测试,比如Syslog、log、CSV生成、Json、MySQL注入、RPC写、NetCat等;
- databand-mock-mq:日志模拟工具,通过MQ写产生大量的日志数据供调试测试,比如RabbitMQ写、Kafka写等;
- databand-mock-hadoop:大数据日志模拟工具,hdfs和mapreduce相关;
#### 数据采集清洗(采集清洗工程)
![ETL](https://gitee.com/475660/databand/raw/master/files/etl.png "DataBand 数据采集清洗")
- databand-etl-mysql_ods:采集清洗mysql数据比如MySQL到ods临时中间库(包括Redis、Kafka等);
- databand-etl-mysql_olap:采集清洗mysql数据到OLAP数据仓库;
- databand-etl-mysql_hadoop:采集清洗mysql数据到Hadoop分布式存储;
- databand-etl-logfile_ods:采集清洗半结构化日志文件,比如json、xml、log、csv文件数据到ods临时中间库;
- databand-etl-logfile_olap:采集清洗半结构化日志文件数据到OLAP数据仓库;
- databand-etl-logfile_hadoop:采集清洗日志文件数据到Hadoop分布式存储;
- databand-etl-mq_ods:通过MQ消费采集数据,入ods库;
- databand-etl-mq_olap:通过MQ消费采集数据,入OLAP库;
- databand-etl-mq_hadoop:通过MQ消费采集数据,入Hadoop;- databand-ml:数据科学工程;
#### 数据分析作业(定时作业调度工程)
- databand-job-springboot:定时任务作业调度服务,支持shell,hive,python,spark-sql,java jar任务。
- databand-streamjob-springboot:流数据作业,支持kafka数据消费至clickhouse、mysql、es等。
#### 数据分析门户(后端管理和前端展示工程)
- databand-admin-ui:前后端分离的纯前端UI工程,数据展现(目前未开发);
- databand-admin-thymeleaf:后端权限、关系、站点配置管理(前后端不分离,正在开发的版本),基于若依框架;
- databand-admin-api:数据api服务;
- databand-admin-tools:BI工具集;
#### 实时流数据(2021年-9月更新)
- databand-rt-flinkstreaming:flink实时数据流处理。主要是PV、UV,涉及窗口、聚合、延时、水印、统计、checkpoint等基本用法;
- databand-rt-redis:实时处理的一些缓存存储;
- databand-rt-sparkstreaming:spark实时数据流处理,和flink的功能近似,主要structured streaming;
### 愿景目标
3年愿景目标
![愿景目标](https://gitee.com/475660/databand/raw/master/files/vision.png "愿景目标")
### 工程细节说明
#### databand-mock-api (模拟数据源API工程) API模拟工具
- App.java:一个简单的mock控制台程序
[api mock详情介绍](https://www.yuque.com/docs/share/162f64f8-bc27-41c1-818c-57f6a953fab1#JLCoA)
[api mock工程源码](https://gitee.com/475660/databand/tree/master/databand-mock-api)
#### databand-mock-log (模拟数据源生成日志数据工程) 日志模拟工具
目前是简单的控制台小程序,直接运行main即可。
- CsvMock.java:csv文件生成,运行后在"FILE_PATH"定义的文件夹中可找到csv文件:
- LogMock.java:log文件生成,生成路径见配置文件:logback.xml。 win下默认“c:/logs/”,linux 或 mac下路径请自行修改:
- JsonMock.java:json文件生成,在"FILE_PATH"定义的文件夹中可找到json文件:
- XmlMock.java:xml文件生成,在"FILE_PATH"定义的文件夹中可找到json文件:
- RpcMock.java:rpc输出,运行后可以用flume(或filebeat)进行测试,配置文件见:/flumeConf/avro-memory-log.properties:运行脚本: flume-ng agent --conf conf --conf-file /usr/app/apache-flume-1.8.0-bin/avro-memory-log.properties --name a2 -Dflume.root.logger=INFO,console
- SyslogMock.java:syslog(udp)输出,运行后可以用flume(或filebeat)进行测试,配置文件见:/flumeConf/syslog-log.properties:
- TcpMock.java:Tcp输出,运行后可以用flume进行测试,配置文件见:/flumeConf/syslog-log.properties:
- MySQLMock.java:mysql数据生成,通过list<map>键值对形式对数据表进行写操作。
[log mock工程源码](https://gitee.com/475660/databand/tree/master/databand-mock-log)
#### databand-mock-mq (模拟数据源生成日志数据工程) MQ消息模拟生成工具
目前是简单的控制台小程序,直接运行main即可。
- KafkaProducer.java:Kafka消息生成:
- KafkaConsumer.java:Kafka消息消费:
- RabbitMQProducer.java:RabbitMQ消息生成:
- RabbitMQConsumer.java:RabbitMQ消息消费:
[mq mock工程源码](https://gitee.com/475660/databand/tree/master/databand-mock-mq)
## 使用说明
### 数据源日志
类型分为:
- CSV日志,用于批处理,采用UTF-8字符集,每行(\r\n)表示一条记录,每条记录中各个字段的值使用双引号括起来,并使用逗号(,)分隔;
- Kafka 日志,用于流处理,生产者策略性的产生一些有偏移属性的带日期时间数据。
业务:
- a)产品销售日志,采用CSV格式;
- b)节目播出日志,采用CSV格式;
- c)搜索热词日志,采用kafka;
- d)广告播放日志,采用kafka;
数据定义,批处理类型日志,原始数据源为csv,暂时以这两个业务作为批处理数据演示,实际上平台将是与业务无关的,只关注数据流和数据服务。
一、产品销售csv日志:
处理类:**org.databandtech.logmock.ProductSalesCSVLog**
- 1 产品id productId
- 2 产品分类id categoryId
- 3 型号规格 modelId
- 4 颜色 color
- 5 买家id userId
- 6 购买日期 saleDatetime
- 7 购买数量 buyCount
- 8 购买金额 buyTotle
- 9 折扣金额 buyDiscount
- 10 城市 cityCode
- 11 地址 address
二、节目播出csv日志
处理类:**org.databandtech.logmock.ShowsCSVLog**
- 1 用户id userId
- 2 状态类型码 status
- 3 城市 cityCode
- 4 区县 areaCode
- 5 收视开始时间 beginTime
- 6 收视结束时间 endTime
- 7 节目ID showId
- 8 栏目ID columnId
- 9 频道ID channelId
- 10 高清标志码 hd
- 11 节目类型码 showType
状态类型码:
- 1:"tv_playing"、2:"vod_playing"、3:"browsing"、4:"tvod_playing"、5:"ad_event" 、6:"external_link"、7:"order"
高清标志码:
- 0:标清、1:高清、2:智能、3:其他
节目类型码:
- 电视剧:tv、电影:movie、综艺:
没有合适的资源?快使用搜索试试~ 我知道了~
DataBand数据帮 轻量级一站式大数据分析平台 源码.zip
![preview](https://csdnimg.cn/release/downloadcmsfe/public/img/white-bg.ca8570fa.png)
共2000个文件
java:647个
js:360个
vue:247个
![preview-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/scale.ab9e0183.png)
需积分: 1 0 下载量 168 浏览量
2023-03-21
14:28:47
上传
评论
收藏 18.04MB ZIP 举报
温馨提示
DataBand(数据帮),快速采集清洗,数据分析,预测分析,人工智能赋能服务,是一站式的大数据平台。我们致力于通过提供智能应用程序、数据分析和咨询服务来提供最优解决方案。
资源推荐
资源详情
资源评论
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![txt](https://img-home.csdnimg.cn/images/20210720083642.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![txt](https://img-home.csdnimg.cn/images/20210720083642.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
收起资源包目录
![package](https://csdnimg.cn/release/downloadcmsfe/public/img/package.f3fc750b.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/DOCX.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/DOCX.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/DOCX.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/UNKNOWN.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/EXE.png)
![file-type](https://csdnimg.cn/release/download/static_files/pc/images/minetype/EXE.png)
共 2000 条
- 1
- 2
- 3
- 4
- 5
- 6
- 20
资源评论
![avatar-default](https://csdnimg.cn/release/downloadcmsfe/public/img/lazyLogo2.1882d7f4.png)
![avatar](https://profile-avatar.csdnimg.cn/f0a931ab09784fa5977bd59ab9d50d86_csdn_manong1.jpg!1)
程序猿小乙
- 粉丝: 63
- 资源: 1740
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![voice](https://csdnimg.cn/release/downloadcmsfe/public/img/voice.245cc511.png)
![center-task](https://csdnimg.cn/release/downloadcmsfe/public/img/center-task.c2eda91a.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![dialog-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/green-success.6a4acb44.png)