没有合适的资源?快使用搜索试试~ 我知道了~
Uma:HDFS Name Node高可用性研究
需积分: 0 0 下载量 115 浏览量
2014-05-29
14:05:37
上传
评论
收藏 941KB PDF 举报
温馨提示
试读
21页
In this presentation an alternative and robust Name Node High Availability implementation based on Bookkeeper is explained. The presentation summarizes the motivation, solution approach, implementation notes and test results.
资源详情
资源评论
资源推荐
HUAWEI TECHNOLOGIES CO., LTD.
Slide title :40-47pt
Slide subtitle :26-30pt
Color::white
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers and
partners :
Arial
www.huawei.com
Namenode High Availability
Uma Maheswara Rao G
maheswara@huawei.com
umamahesh@apache.org
HUAWEI TECHNOLOGIES CO., LTD.
Slide title :32-35pt
Color: R153 G0 B0
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Slide text :20-22pt
Bullets level 2-5:
18pt
Color:Black
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Top right corner
for field-mark,
customer or partner
logotypes.
----------------
The following nine
groups of colors
are an example of
how our design
colors can be used,
please take note
that you should
only use one
design color group
per slide.
For specific usage
details, refer to the
“Typesetting
Standard”.
Page 2
Who am I
R&D Tech Lead in Huawei
Apache Hadoop Committer at ASF
Active Contributor to Apache BookKeeper at ASF
Main development focus on HDFS
NameNode HA
HUAWEI TECHNOLOGIES CO., LTD.
Slide title :32-35pt
Color: R153 G0 B0
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Slide text :20-22pt
Bullets level 2-5:
18pt
Color:Black
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Top right corner
for field-mark,
customer or partner
logotypes.
----------------
The following nine
groups of colors
are an example of
how our design
colors can be used,
please take note
that you should
only use one
design color group
per slide.
For specific usage
details, refer to the
“Typesetting
Standard”.
Page 3
Huawei Hadoop R&D – In a Glance
• Secondary Index in HBase
• HDFS NN HA (Hadoop-2)
• Bookkeeper as shared storage for NN HA (Hadoop-2)
• HDFS NN HA (Hadoop-1)
• MapReduce ResourceManager HA (Hadoop-2 / YARN)
• MapReduce JobTracker HA (Hadoop-1)
• Hive HA
Hadoop Development
• Raised over 650 defects since Jan’11
• Fixed over 500 defects since Jan’11, and contributed back
to community.
Stabilization
NameNode HA
HUAWEI TECHNOLOGIES CO., LTD.
Slide title :32-35pt
Color: R153 G0 B0
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Slide text :20-22pt
Bullets level 2-5:
18pt
Color:Black
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Top right corner
for field-mark,
customer or partner
logotypes.
----------------
The following nine
groups of colors
are an example of
how our design
colors can be used,
please take note
that you should
only use one
design color group
per slide.
For specific usage
details, refer to the
“Typesetting
Standard”.
Page 4
In 2011, we implemented HA in Hadoop 0.20.1, based on
Backup Namenode(BNN) and ZooKeeper at Huawei
Intelligent clients - find active NN from configured NNs
Streaming edits to the BNN
Sending block reports to both active NN and BNN
BNN does periodic checkpoints
ZK based leader election
Achieved hot standby and Automatic Switch
However, BNN based solution did not address double failure
scenario
Namenode High Availability
NameNode HA
HUAWEI TECHNOLOGIES CO., LTD.
Slide title :32-35pt
Color: R153 G0 B0
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Slide text :20-22pt
Bullets level 2-5:
18pt
Color:Black
Corporate Font :
FrutigerNext LT Medium
Font to be used by customers
and
partners :
Arial
Top right corner
for field-mark,
customer or partner
logotypes.
----------------
The following nine
groups of colors
are an example of
how our design
colors can be used,
please take note
that you should
only use one
design color group
per slide.
For specific usage
details, refer to the
“Typesetting
Standard”.
Page 5
What is Double Failure here?
Active NN
Backup NN
Active NN Start - Active NN
Checkpoint – t1
Checkpoint – t2
NN-1
NN-2
NN-1 NN-2
X - Operation logs in NN1 and could not stream to
BNN (NN2)
Not aware of X - Operation logs
from NN1. DataLoss!!!
NN1 is active and NN2 is Backup node
NN1 failed to stream edits to the Backup node and removed the stream
NN1 received X- operation edit logs, where NN2 not aware of them
Now NN1 crashed, NN2 becomes active
NN2 continues to work but not aware of the edits after checkpoint:t1
NameNode HA
剩余20页未读,继续阅读
admin
- 粉丝: 43
- 资源: 314
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
评论0