没有合适的资源?快使用搜索试试~ 我知道了~
Howtoovercomemysteriousproblemscausedbylargeandmultitenanthadoop...
需积分: 6 0 下载量 131 浏览量
2019-08-28
20:50:39
上传
评论
收藏 1.6MB PDF 举报
温馨提示
在Hadhoop summit 2016上,Tomomichi Hirano ,EC Core Technology Department, Rakuten Inc分享了题为《How to overcome mysterious problems caused by large and multi-tenant hadoop cluster at Rakuten》,就Rakuten的Hadoop用例,大数据系统的背景下的服务器配置和管理等方面的内容做了深入的分析。
资源推荐
资源详情
资源评论
How to overcome mysterious problems
caused by large and multi-tenant
hadoop cluster at Rakuten
Oct/27/2016
Tomomichi Hirano
EC Core Technology Department, Rakuten Inc.
tomomichi.hirano@rakuten.com
Who I am
n Tomomichi Hirano (平野 智巌)
• Joined to Rakuten in 2013
• Hadoop administrator
• Monitoring, tuning and Improving hadoop cluster
• Verifying and enabling new hadoop related components
• Trouble shooting for all problems
• Regular operation such as server and user adding, disk replacement, etc.
n Previous team
• Server provisioning, networking and HW related things.
2
Today’s Agenda
n Quick introduction
• About our clusters
• Hadoop use cases at Rakuten
n Mysterious problems
• Never ending jobs
• DataNode freezing
• NameNode freezing
• High load after restarting NameNode
• Lessons learned
3
n Server provisioning and
management
• Background for Big Data systems
• Provisioning and management
4
1
Quick introduction
About our clusters
n Production cluster
• # of slaves : around 200
• HDFS capacity : around 8PB
• # of jobs per day : 30,000 - 50,000
• # of hadoop active user accounts : around 40
• Types of jobs : MR, Hive, Tez, Spark, Pig, sqoop, HBase, Slider, etc.
n Other clusters
• Another production cluster for Business Continuity (BC).
• Some clusters for staging and development.
5
剩余53页未读,继续阅读
资源评论
weixin_38743602
- 粉丝: 396
- 资源: 2万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功