spark源码阅读笔记(详)

Spark 源码阅读笔记

目录

 源码阅读笔记 

 源码之  与  

 节点的启动过程分析

 节点启动和注册

 进程的启动 

 节点主要流程 

 !! 

 !! 分区列表。 

 !! 的依赖列表 

窄依赖 

宽依赖 

 !! 的分区计算函数 "

分区器 

位置偏好 

全排序 

水塘采样 

"'(&) 的启动

"* 的注册 

"资源调度 

"& 的启动 "

 的分发 

(& 初始化 

&%+#, "

&-. "

, 的调度 "

任务本地性 "

(&/+,0(/1 

2((&/+# 运行原理

(&/+# 运行原理

 的执行（二） "

(&/+,0(/  

345 网络传输服务 

"(&/ 相关参数调优

 &, 

存储模块 

)+# 初始化 

架，利用这个框架我们可以非常容易的开发并行程序，然后在集群上运行，

spark 社区为我们提供了 Java，Scala，python,R 语言等很多接口，每个程

序员可以选择自己熟悉的语言进行开发。

spark 在实现过程中，利用了很多现成的技术和思想，例如通信框架选用

的是 akka 和 netty,这两种通信技术方案已经在工业界盛行多年，有很多成功

的案例可以借鉴；spark she 的实现 sortshe 的实现其实是完全借鉴了

mapreduce she 实现的思路，只是实现细节不太一样；spark 在底层调度

的过程中也尽可能的复用了 yarn 和 mesos 这两种的资源调度机制。

可以这样认为 spark 其实是 hadoop 的加强版，他们同根同源利用的都是

mapreduce 这种先切分后聚合的思想，只是 hadoop 对于数据处理的思路单

样的话整个 job 过程只需要在 Hdfs 读写一次，不需要像 mapreduce

那样在 hdfs 中间写入多次，大大提高了速度。

 申请资源方式不同。spark 上 Excutor 对应一个 Jvm 资源，多个 task

对应 Jvm 上的多个线程，Excutor 可以被多个 task 所复用，申请资源

次数比较少。Mapreduce 每一个 task 任务会对应一个进程，且相互

之间不能复用，申请资源次数比较多。

 RDD 缓存机制。spark 对于中间运算的结果可以缓存在内存，下次再

取数据时就不必重新计算，这非常适用于一些迭代式的任务的运行，

可以大大提高速度。

 容错机制。spark 在 DAG 的基础上建立了每个 RDD 的血统依赖和检

mesos 版本的基础。

spark 后续版本可能在某些细节上有所优化，增加了一些库文件，但基本

思想并未改变，所以下文引入的源码基本上都是 spark 1.2 的源码。