CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together

5星(超过95%的资源)
所需积分/C币:9 2016-05-16 18:32:16 1.32MB PDF
27
收藏 收藏
举报

该文档来自CCTC 2016中国云计算技术大会。Hortonworks技术专家邵赛赛发表的题为“Spark and YARNBetter Together”的主题演讲,欢迎下载!
Overview of Spark Cluster Executor Executor Executor Cluster Manager Driver O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS Spark Running on YARN(Client Mode) ResourceManager NM NM NM Executor Executor Executor AM Driver 4 O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS Spark Running On YARN(Cluster Mode ResourceManager NM NM NM Executor Executor Executor AM Client Driver O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS Difference compared to other cluster Manager e Application has to be submitted into a queue JArs /files/archives are distributed through distributed cache e Additional application Master O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS Better Run Spark On YARN O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS What Do We Concern about e Better use the resources ● Better run on c| uster ● Easy to debug O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS Calculate Container size spark executor. memory e What is the size of a container spark memory fraction(0.75 Memory spark. memory. storageFr execution action memory CPU 05 sparkyarn executor. memoryOverhead (offheap spark. memory. offHeap size container memory =spark executor memory overhead memory yarn scheduler. minimum-allocation-mb < container memory < yarn nodemanager. resource. memory-mb container memory will be round to yarn scheduler. increment-allocation-mb O Hortonworks Inc 2011-2016. All Rights Reserved Depend on Cpu scheduling is enabled or not HORTONWORKS Calculate Container Size(Contd) Node a-8 cores. 64 GB o Enable CPU Scheduling Capacity Scheduler with Default Resource Calculator(default 8 GB Only takes memory into account CPU requirements are ignored when carrying out 4 cores allocations The setting of -executor-cores"is controlled by spark 32 GB itself Capacity Scheduler with DominantResource Calculator CPU Will also take into account when calculating 3 cores Container vcores= executor cores 16 GB container cores < nodemanger.resource.cpu-vcores 1 core 10 O Hortonworks Inc 2011-2016. All Rights Reserved HORTONWORKS

...展开详情
试读 23P CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together
立即下载 身份认证后 购VIP低至7折
一个资源只可评论一次,评论内容不能少于5个字
b2bsssss 大神出品,深入浅出
2018-02-16
回复
您会向同学/朋友/同事推荐我们的CSDN下载吗?
谢谢参与!您的真实评价是我们改进的动力~
  • 分享王者

关注 私信
上传资源赚钱or赚积分
最新推荐
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together 9积分/C币 立即下载
1/23
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together第1页
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together第2页
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together第3页
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together第4页
CCTC 2016 Hortonworks邵赛赛:Spark and YARNBetter Together第5页

试读结束, 可继续读2页

9积分/C币 立即下载