【spark2.4.4源码编译】windows环境编译spark2.4.4源码windows环境编译spark2.4.4源码
环境要求
操作系统环境：Windows 10（Windows7、Windows8亦可）
Java版本： jdk1.8
Scala版本：2.11.0
Maven版本：3.5.4
Git版本：版本⽆要求
以上相关组件的版本是根据spark2.4.4源码的pom⽂件⾥的组件版本进⾏梳理的：
<java.version>1.8</java.version>
<maven.version>3.5.4</maven.version>
<scala.version>2.11.12</scala.version>
根据这组配置，本⼈已经成功编译数遍。
环境安装
1、Git Bash
⾸先说明，为什么要装Git Bash这个东西，这个主要是为了⽅便我们能在windows环境下执⾏linux环境下的⼀些操作。下载链接：
git-scm/download/win
根据操作系统的位数下载相应的安装包之后，进⾏安装就可以了。
2、JDK1.8
下载链接：
acle/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
3、Scala2.11.0
下载链接：
/download/2.11.0.html
mirrors.tuna.tsinghua.edu/apache/maven/maven-3/3.5.4/binaries/
以上下载安装之后，在系统变量⾥配置相关环境变量：
最后打开Git（或者CMD）进⾏相关验证：
Jack_Roy@LAPTOP-FAAF41TU MINGW64 ~
$ mvn -version
Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-18T02:33:14+08:00)
Maven home: D:\apache-maven-3.5.4
Java version: 1.8.0_181, vendor: Oracle Corporation, runtime: D:\JAVA\JDK\jre
Default locale: zh_CN, platform encoding: GBK
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
Jack_Roy@LAPTOP-FAAF41TU MINGW64 ~
$ java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
Jack_Roy@LAPTOP-FAAF41TU MINGW64 ~
$ scala -version
Scala code runner version 2.11.0 -- Copyright 2002-2013, LAMP/EPFL
以上就是编译环境的全部准备⼯作了，如果你的java、scala/maven统统验证成功，那么就可以下载spark源码进⾏编译了。源码下载
spark2.4.4源码下载链接：
/downloads.html
在下载页⾯选择相应版本和安装包类型（source code）后，就可以进⾏下载了：
源码编译
下载完成后我们进⾏解压，然后在Git中到下载路径：
$ cd d:
Jack_Roy@LAPTOP-FAAF41TU MINGW64 /d
$ cd spark_source/spark-2.4.4
Jack_Roy@LAPTOP-FAAF41TU MINGW64 /d/spark_source/spark-2.4.4 $ ll
total 220
-rw-r--r-- 1 Jack_Roy 197121 2298 8⽉ 28 05:l
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 assembly/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 bin/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 build/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 common/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 conf/
-rw-r--r-- 1 Jack_Roy 197121 995 8⽉ 28 05:20 CONTRIBUTING.md drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 core/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 data/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 dev/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 docs/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 examples/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 external/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 graphx/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 hadoop-cloud/ drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 launcher/
-rw-r--r-- 1 Jack_Roy 197121 13335 8⽉ 28 05:20 LICENSE
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 licenses/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 mllib/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 mllib-local/
-rw-r--r-- 1 Jack_Roy 197121 1531 8⽉ 28 05:20 NOTICE
-rw-r--r-- 1 Jack_Roy 197121 103442 8⽉ 28 05:l
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 project/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 python/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 R/
-rw-r--r-- 1 Jack_Roy 197121 3952 8⽉ 28 05:20 README.md
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 repl/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 resource-managers/ drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 sbin/
-rw-r--r-- 1 Jack_Roy 197121 18386 8⽉ 28 05:l drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 sql/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 streaming/
drwxr-xr-x 1 Jack_Roy 197121 0 11⽉ 9 13:54 tools/
输⼊编译命令执⾏：
mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
如下就是开始编译了：
$ mvn -Pyarn -Phadoop-2.4-Dscala-2.11-DskipTests clean package
[INFO] Scanning
Downloading from alimaven: maven.aliyun/nexus/content/groups/public/org/apache/apache/18/apache-18.pom
Downloaded from alimaven: maven.aliyun/nexus/content/groups/public/org/apache/apache/18/apache-18.pom(16 kB at 14 kB/s) [INFO]------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM[pom]
[INFO] Spark Project Tags [jar]
[INFO] Spark Project Sketch [jar]
[INFO] Spark Project Local DB[jar]
[INFO] Spark Project Networking [jar]
[INFO] Spark Project Shuffle Streaming Service [jar]
[INFO] Spark Project Unsafe [jar]
[INFO] Spark Project Launcher [jar]
[INFO] Spark Project Core [jar]
[INFO] Spark Project ML Local Library [jar]
[INFO] Spark Project GraphX [jar]
[INFO] Spark Project Streaming [jar]
[INFO] Spark Project Catalyst [jar]
[INFO] Spark Project SQL[jar]
[INFO] Spark Project ML Library [jar]
[INFO] Spark Project Tools [jar]
[INFO] Spark Project Hive [jar]
[INFO] Spark Project REPL[jar]
[INFO] Spark Project YARN Shuffle Service [jar]
[INFO] Spark Project YARN[jar]
下载apache[INFO] Spark Project Assembly [pom]
[INFO] Spark Integration for Kafka 0.10[jar]
[INFO] Kafka 0.10+ Source for Structured Streaming [jar]
[INFO] Spark Project Examples [jar]
[INFO] Spark Integration for Kafka 0.10 Assembly [jar]
[INFO] Spark Avro [jar]
[INFO]
[INFO]-----------------< org.apache.spark:spark-parent_2.11>-----------------
[INFO] Building Spark Project Parent POM2.4.4[1/26]
[INFO]--------------------------------[ pom ]---------------------------------
⼀个半⼩时后：
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论