Spark 源码编译完整教程（maven编译+sbt编译+编译工具包+操作步骤笔记+依赖包下载镜像设置）

1.指定依赖包的mirror有两种方式： 1）可以在project的 pom.xml中指定依赖包的mirror，就像这样： <project> … <repositories> <repository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </pluginRepository> </pluginRepositories> … </project> 说明： a) <releases><enabled>true</enabled></releases>告诉Maven可以从这个仓库下载releases版本的构件； b) <snapshots><enabled>false</enabled></snapshots>告诉Maven不要从这个仓库下载snapshot版本的构件； c) <repositories>配置Maven从什么地方下载项目的依赖包； d）<pluginRepositories>配置Maven从什么地方下载插件构件，Maven的所有行为都是通过插件来完成的，其内部配置与<repository>类似； e) 使用这种方式，只适用于当前项目. 2）修改 $MAVEN_HOME/conf/settings.xml <profiles> <profile> <id>jdk-1.7</id> <activation> <jdk>1.7</jdk> </activation> <repositories> <repository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </repository> </repositories> <pluginRepositories> <pluginRepository> <id>nexus</id> <name>local private nexus</name> <url>http://maven.oschina.net/content/groups/public/</url> <releases> <enabled>true</enabled> </releases> <snapshots> <enabled>false</enabled> </snapshots> </pluginRepository> </pluginRepositories> </profile> </profiles> 说明： a)该配置将应用到所有项目，且会覆盖应用的配置； b)该配置需要激活，此处是使用JDK的版本来自动激活，如果要强制激活，使用 <activeProfiles> 标签 2.如果没有配置下载库，则默认使用中央库：https://repo1.maven.org/maven2/org/， 该库的地址可以通过$MAVEN_HOME/conf/settings.xml修改 <mirrors> <mirror> <id>nexus-osc</id> <mirrorOf>central</mirrorOf> <name>Nexus osc</name> <url>http://maven.oschina.net/content/groups/public/</url> </mirror> <mirror> <id>nexus-osc-thirdparty</id> <mirrorOf>thirdparty</mirrorOf> <name>Nexus osc thirdparty</name> <url>http://maven.oschina.net/content/repositories/thirdparty/</url> </mirror> </mirrors> 说明：以上修改了中央库 central的URL, 同时增加了一个第三方库供选择。 参考网址：http://www.blogjava.net/baoyaer/articles/355040.html 3.重定向镜像后，编译结果如下： [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [03:16 min] [INFO] Spark Launcher Project ............................. SUCCESS [01:50 min] [INFO] Spark Project Networking ........................... SUCCESS [ 23.532 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.690 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 7.697 s] [INFO] Spark Project Core ................................. SUCCESS [08:22 min] [INFO] Spark Project Bagel ................................ SUCCESS [ 17.952 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 46.688 s] [INFO] Spark Project Streaming ............................ SUCCESS [01:19 min] [INFO] Spark Project Catalyst ............................. SUCCESS [01:38 min] [INFO] Spark Project SQL .................................. SUCCESS [02:00 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:50 min] [INFO] Spark Project Tools ................................ SUCCESS [ 5.547 s] [INFO] Spark Project Hive ................................. SUCCESS [02:29 min] [INFO] Spark Project REPL ................................. SUCCESS [ 22.626 s] [INFO] Spark Project YARN ................................. SUCCESS [ 34.331 s] [INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 30.883 s] [INFO] Spark Project Assembly ............................. SUCCESS [03:22 min] [INFO] Spark Project External Twitter ..................... SUCCESS [ 17.781 s] [INFO] Spark Project External Flume Sink .................. SUCCESS [ 36.825 s] [INFO] Spark Project External Flume ....................... SUCCESS [ 21.022 s] [INFO] Spark Project External MQTT ........................ SUCCESS [01:14 min] [INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 17.010 s] [INFO] Spark Project External Kafka ....................... SUCCESS [ 29.272 s] [INFO] Spark Project Examples ............................. SUCCESS [04:07 min] [INFO] Spark Project External Kafka Assembly .............. SUCCESS [01:02 min] [INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 12.078 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 39:07 min [INFO] Finished at: 2015-08-20T18:46:57+08:00 [INFO] Final Memory: 93M/904M [INFO] ------------------------------------------------------------------------