超级详细的spark体系思维导图

<?xml version="1.0" encoding="UTF-8" standalone="no"?><xmap-content xmlns="urn:xmind:xmap:xmlns:content:2.0" xmlns:fo="http://www.w3.org/1999/XSL/Format" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink" modified-by="ganling" timestamp="1542074803840" version="2.0"><sheet id="5gimpj6r68ai2llsfdlm0l87t0" modified-by="ganling" theme="6rk3ges1df2vvfvu7ek61ff89o" timestamp="1542074803840"><topic id="2vkscnetvja10ilvnun19mqq1u" modified-by="Administrator" structure-class="org.xmind.ui.map.unbalanced" timestamp="1516073856960"><title>SparkCore</title><children><topics type="attached"><topic id="4f7u44fp19t09vmbj2nglrj90b" modified-by="ganling" timestamp="1542073656646"><title>前言</title><children><topics type="attached"><topic id="4diakcgg7q2njbbojf3o57ubap" modified-by="ganling" timestamp="1542073657625"><title>对比MapReduce</title><children><topics type="attached"><topic id="088rbc99md6ohscnfdmb5kuj7t" modified-by="ganling" timestamp="1542073658862"><title>分布式计算框架</title><children><topics type="attached"><topic id="5f84c0b89dee80m3iap3hq0g1p" modified-by="ganling" timestamp="1542073669179"><title>就MapReduce单个而言它只是一个计算框架，没有什么环境的</title><children><topics type="attached"><topic id="0rklreqiieehv4sjfs4u8jcc81" modified-by="ganling" timestamp="1542073677953"><title>只是实际工作的时候把它运行在yarn上面</title><children><topics type="attached"><topic branch="folded" id="7bmsfg95d51oegui49ada7r3e6" modified-by="admin" timestamp="1525836616641"><title>MapReduce需要读取HDFS文件所以会结合HDFS</title></topic></topics></children></topic></topics><topics type="callout"><topic id="1g70kq5ph35pj8e32u8qeod9ik" modified-by="Administrator" timestamp="1509017594562"><title>就和Spring MVC一样</title><position svg:x="0" svg:y="0"/><children><topics type="attached"><topic id="0blemm3qn1ujiui1bsab39p3in" modified-by="Administrator" timestamp="1509017594562"><title>spring运行在tomcat上</title></topic></topics></children></topic></topics></children></topic></topics></children></topic><topic id="4lifdmd92nmr9ukpqril1a5i92" modified-by="ganling" timestamp="1542073664889"><title>缺点</title><children><topics type="attached"><topic id="7ume34lhjggqro6jkaisab0huv" modified-by="ganling" timestamp="1542073693874"><title>执行速度慢</title><children><topics type="callout"><topic id="7ongko6ggimk5vb5dngi2afhm1" modified-by="Administrator" timestamp="1509017594563"><title>3.0速度快了很多</title><position svg:x="0" svg:y="0"/></topic></topics><topics type="attached"><topic id="2l1t4pu7v50e70rbq4ss8amjkj" modified-by="ganling" timestamp="1542073695516"><title>IO瓶颈</title><children><topics type="attached"><topic id="7eec6l143qtk1ajsr9aagr4i18" modified-by="ganling" timestamp="1542073696761"><title>磁盘IO</title><children><topics type="attached"><topic branch="folded" id="67tnrhsg8t689nkftd1jvatrae" modified-by="admin" timestamp="1525836616641"><title svg:width="500">每个job的结果需要存储到HDFS集群机器的磁盘，而每个job又至少有三个备份因子，需要写3份，所以很慢</title></topic></topics></children></topic><topic branch="folded" id="3g1bv17t739t50lk92oa2v1fv6" modified-by="admin" timestamp="1525836616641"><title>网络IO</title></topic></topics></children></topic><topic id="5daho8oj6pq2403dvcuobsb9uf" modified-by="ganling" timestamp="1542073699807"><title>shuffle机制</title><children><topics type="attached"><topic id="2ppk1fbqacmc0a6jfd9pa82pvr" modified-by="ganling" timestamp="1542073705277"><title>数据需要输出到磁盘，而且每次都需要进行排序的操作</title><children><topics type="attached"><topic id="46hpkgpd51rnkpp475fc46jaj6" modified-by="ganling" timestamp="1542073706574"><title>当文件比较大的时候它执行的是外排序</title><children><topics type="callout"><topic id="79vhsjcupdq3svuh40evi2kd8b" modified-by="Administrator" timestamp="1509017594565"><title>外排序：在磁盘上存取文件排序</title><position svg:x="0" svg:y="0"/></topic></topics><topics type="attached"><topic branch="folded" id="3l5fhknfvtri8oofqnitoe7tf1" modified-by="admin" timestamp="1525836616641"><title>外排序是比较慢的</title><children><topics type="callout"><topic id="25amb2tjclarhj2ioggp0dm775" modified-by="Administrator" timestamp="1509017594567"><title>实质也是内存利用率方面的问题</title><position svg:x="0" svg:y="0"/><children><topics type="attached"><topic id="5ui4h5i2vqkejd6ss3g9in8hhv" modified-by="Administrator" timestamp="1509017594566"><title>MapReduce的maptask默认是分配1个g的内存</title><children><topics type="attached"><topic id="7k4rahr4ss76p4redkngjfmkdo" modified-by="Administrator" timestamp="1509017594566"><title>实质上MaptaskJVM的大小才200M</title><children><topics type="attached"><topic id="2c3qh7m1fvbfbbgrk7pjo4q21i" modified-by="Administrator" timestamp="1509017594566"><title>当输出到shuffle的时候，shuffle的缓冲区才100MB</title><children><topics type="attached"><topic id="33vvn5h60th85uridh3u590s0h" modified-by="Administrator" timestamp="1509017594566"><title>它希望剩余的100MB程序中用</title></topic></topics></children></topic></topics></children></topic></topics></children></topic><topic id="1eqm9s2snnnas4epurlu65to54" modified-by="Administrator" timestamp="1509017594567"><title>如果一个task分配500MB</title><children><topics type="attached"><topic id="32s4208em8famqrh9p5s8j3e45" modified-by="Administrator" timestamp="1509017594567"><title>shuffle分配400MB</title><children><topics type="attached"><topic id="0mkbcmnqm6l53stvl97mlun4p6" modified-by="Administrator" timestamp="1509017594567"><title>其它程序分配100MB</title><children><topics type="attached"><topic id="551k6f9qbtcsbasfrnoabs9gne" modified-by="Administrator" timestamp="1509017594566"><title>如果数据不太大的话就可以在内存中处理完。不用到磁盘中读写</title><children><topics type="attached"><topic id="6p52atmm0gqdo65hfhjcovom51" modified-by="Administrator" timestamp="1509017594566"><title>因为一个Maptask才处理128M数据</title></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic></topics></children></topic><topic id="7cl4214p8vup1sati32cqcaslk" modified-by="ganling" timestamp="1542073709990"><title>框架的缺陷</title><children><topics type="attached"><topic id="204fv4i1r474j1qs0ubpklj04q" modified-by="ganling" timestamp="1542073711719"><title svg:width="500">只有map和reduce两个操作算子，结构只能是 map -&gt;reduce 或者map -&gt;[map -&gt;] *-&gt;reduce[-&gt;map]*</title><children><topics type="attached"><topic branch="folded" id="4p8k765j14sn5ugg5gg77bct35" modified-by="admin" timestamp="1525836616641"><title svg:width="500">针对比较复杂的任务，需要构建多个job来执行，而多个job有依赖关系的时候，它会增加数据的读写、网络数据传输等方面的问题</title></topic></topics></children></topic></topics></children></topic><topic id="0vbjaf14k8f0fmvm7r385ejaho" modified-by="ganling" timestamp="1542073724215"><title>进程方式启动</title><children><topics type="attached"><topic id="7ar2b2k9anncbj0r3g2or6m28j" modified-by="ganling" timestamp="1542073725532"><title>Task是以进程的方式来启动的</title><children><topics type="attached"><topic id="545ndv66cf73v2fep7pjau78u0" modified-by="ganling" timestamp="1542073727597"><title>你会发现每个MR运行的时候，前面总会占了几十秒</title><children><topics type="attached"><topic id="3c188evm84djjl7gde4o0ioa55" modified-by="ganling" timestamp="1542073730982"><title>那就是起进程的过程</title><children><topics type="attached"><topic b