没有合适的资源?快使用搜索试试~ 我知道了~
一个spark streaming+Cassandra、分布式作业调度系统总结
5星 · 超过95%的资源 需积分: 50 62 下载量 24 浏览量
2017-03-10
20:27:41
上传
评论 4
收藏 4.51MB PDF 举报
温馨提示
试读
29页
一个项目的总结文档,英文,主要功能: 1、分布式作业管理、调度、监控; 2、基于spark streaming+Cassandra的实时分析和监控,包括性能分析、账号安全主动防御。 web部分采用spring boot开发,前端采用angularJS组织页面相关的各个部分,系统的技术和效果在2016~2017年在行业具有一定先进性。
资源推荐
资源详情
资源评论
ShenCe Introduction
Dec. 2016
ShenCe Introduction
1
Web R&D’s New Plays
2
Big Data R&D’s New Plays
3
General Introduction
l Job Scheduling: GUI, cluster based, dynamic, distributed, relible job mgmt & scheduling service
l Intrusion Detection: User account & data protection; Account platform protection, 3 levels
l Real-time Monitor:Access count, success rate, performance of any Nginx based service , 3 levels
Login
Data
Intrusion
Detection
Real-time
Monitor
Job
Scheduling
ShenCe Web
Data Collect
Data Analysis
Job Dispatch
Real-time Act
GUI
Automatic and Intelligent Platform
Web Server
Log
Distributed
Scripts
p Main Advantages
ü Good commonality (Nginx, Apache, …)
ü No any influence to online service
ü Good real-time (Not more than 5 seconds)
ü 3 levels monitor (Cluster -> URL -> IP)
p Current Status
ü Has been deployed in production environment
ü Scheduling jobs in 3 machine rooms
ü Real-time detect, notify to intercept IP & protect
data
Resources(&(Progress
Jul. Aug.%~%Sep. Oct.%~%Nov. Dec.
l Evaluation,
planning from
middle July
l 6 interns on July 20
l 2 interns and I
started developing
job scheduling part
on July 23
l Finish job related
R&D
l Deployed in cloud,
schedule job in
two MRs
l Begin focus on
spark related parts
in Sep.
l 7 people joined in
Sep., 1 internal
l Set up R&D ENV with
CLI + Maven
l Finish spark related
R&D and optimization
l Deployed in cloud &
physical machine ENV
l Connect to other
services in cloud ENV
l 1 of 3 people 100%
joined on Oct. 27
l Break through
limitations of CN2
in proxy
l Connect with other
services in
PHYSICAL ENV
l Requirements
adjusting & adding
l 1 intern joined on
Dec. 21
Total Human Resources:
R&D: 17 in all, 8 interns, 2 formal employees, 7 part time employees
Operations: 3 employees, part time
3 Spark Apps
Total Memory: 80Gb~120Gb
NginxLogAnalyzer
PopLoginAnalyzer
WebLoginAnalyzer
剩余28页未读,继续阅读
资源评论
- Samanoseky2017-08-23挺好的, 写的很详细. 可以作为自己开发相关产品时候的参考.
tyq1
- 粉丝: 2
- 资源: 29
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功