CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践

5星(超过95%的资源)
所需积分/C币:9 2016-05-17 11:35:32 1.27MB PDF
25
收藏 收藏
举报

该文档来自CCTC 2016中国云计算技术大会。Hulu大数据架构组负责人董西成发表的题为“Hadoop YARN在异构环境下应用与实践”的主题演讲,欢迎下载!
hulu ARCHITECTURE YARN MapReduce Spark Nesto/Pr Voidbox (Streaming esto YARN distributed operating system machine machine machine hul ARCHITECTURE YARN Node Manager Container Container (Map Task) Client NodeManager Applicatio Container e source (Map Task) n Manager Maste Client Hdfs NodeManager MR AppMast HDFS DISTRIBUTED APPLICATION TYPES Batch Jobs Take from a few seconds to a few days to complete E.g. MapReduce, Spark Long-running Services Should "never go down, handle short-lived latency-sensitive requests E.g. Presto/Nesto, Spark Streaming, Web Services YArN: Data Operating system ResourceManager ha recovery N odeManager Recovery Resource Scheduling(e. g. label-based scheduling hulu ° Apache Slider& Apache Tw11一 HETEROGENEOUS ISSUES Heterogeneous causes Static factors: Machines(CPU, memory, network, OS, libraries) Dynamic factors: Load, Network, Slow Disk Solutions for static factors Distinguish machines, and assign to different frameworks E.g. 128GB-memory machines for spark, 10G -network machines for spark streaming/nesto/ presto Solutions for dynamical factors Solve in framework level For MapReduce /Spark: speculative execution For others hulu AGENDA Background Label-based scheduling nul Framework On YARN Experience Q& a WHY LABEL- BASED SCHEDULING Distinguish good/bad machines for different Apps E.g. Large memory machine for spark Apps require special libraries/services installed Spark mllib: native libraries Voidbox: docker engines Machine-level isolation for key apps Low-latency Apps: spark streaming/presto/nesto hulu LABEL- BASED SCHEDULING roo node node Resource manager node ( Capacity Scheduler node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node node hulu LEARN FROM LABEL-BASED SCHEDULING Benefit Static/ dynamic machine-level partition Elastic scalability for Apps Latch to enter maintenance mode .Issue Only capacity scheduler is supported YARN Web Portal does not support well Hurt data locality hulu

...展开详情
试读 28P CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践
立即下载 身份认证后 购VIP低至7折
一个资源只可评论一次,评论内容不能少于5个字
FantuanTech 内容不多,但是蛮有参考意义。
2016-10-02
回复
dingdong0080 2016中国云计算技术大会演讲嘉宾PPT
2016-08-15
回复
您会向同学/朋友/同事推荐我们的CSDN下载吗?
谢谢参与!您的真实评价是我们改进的动力~
  • 签到王者

  • 技术圈认证(专家版)

  • 分享王者

关注 私信
上传资源赚钱or赚积分
最新推荐
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践 9积分/C币 立即下载
1/28
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第1页
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第2页
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第3页
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第4页
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第5页
CCTC 2016 Hulu董西成:Hadoop YARN在异构环境下应用与实践第6页

试读结束, 可继续读3页

9积分/C币 立即下载