一 说明
1.1 基础环境
hadoop3.2.1集群已经安装完成
Hadoop 3.2.1 集群
JDK 1.8
1.2 下载Spark 软件包
官方网址如下：
http://spark.apache.org/downloads.html
1.3 下载scala 软件包
官方网址如下：
https://www.scala-lang.org/
二 安装scala
2.1 解压scala 到指定目录
[root@hadoop1 hadoop]# mkdir /hadoop/scala
上传scala-2.13.3.tgz 到/hadoop/soft
tar -xvf scala-2.13.3.tgz -C /hadoop/scala/
2.2 修改环境变量
修改/etc/profile, 添加相应的配置信息
[root@hadoop1 scala-2.13.3]# vi /etc/profile
#set scala environment
export SCALA_HOME=/hadoop/scala/scala-2.13.3
export PATH=$PATH:${JAVA_PATH}:$SCALA_HOME/bin
生效环境变量
[root@hadoop1 scala-2.13.3]# source /etc/profile
2.3 测试scala 配置是否正常
依次安装各个节点。
三 安装配置spark
3.1 上传并解压spark 到spark 安装目录
tar -xvf spark-3.0.0-bin-hadoop3.2.tgz -C
3.2 修改环境变量
修改/etc/profile, 添加相应的配置信息
[root@hadoop1 scala-2.13.3]# vi /etc/profile
#set spark environment
export SPARK_HOME=/hadoop/spark/spark-3.0.0-bin-hadoop3.2
export SPARK_EXAMPLES_JAR=$SPARK_HOME/examples/jars/spark-examples_2.12-3.0.0.jar
export PATH=$PATH:${JAVA_PATH}:$SCALA_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin
生效环境变量
[root@hadoop1 scala-2.13.3]# source /etc/profile
3.3 修改spark-env.sh
[root@hadoop1 ~]# cd /hadoop/spark/spark-3.0.0-bin-hadoop3.2/conf/
[root@hadoop1 conf]# cp spark-env.sh.template spark-env.sh
[root@hadoop1 conf]# vi spark-env.sh
export SCALA_HOME=/hadoop/scala/scala-2.13.3
export JAVA_HOME=/hadoop/jdk1.8/jdk1.8.0_211-amd64
export SPARK_MASTER_IP=hadoop1
# 设置web 页面端口
export SPARK_MASTER_WEBUI_PORT=8888
# Spark master 和worker 守护进程的JVM 选项（默认：none)
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=hadoop1:2181,hadoop2:2181,hadoop3:2181 -Dspark.deploy.zookeeper.dir=/spark"
3.4 修改slaves
( 在其中加入集群的从节点的主机或者IP)
[root@hadoop1 conf]# cp slaves.template slaves
[root@hadoop1 conf]# vi slaves
hadoop2
hadoop3
3.5 将spark 拷贝到其他节点
[root@hadoop1 hadoop]# scp -r /hadoop/spark hadoop2:/hadoop/
[root@hadoop1 hadoop]# scp -r /hadoop/spark hadoop3:/hadoop/
3.6 启动spark
首先确保hadoop 集群已经正常启动
[root@hadoop1 soft]# jps
23968 NameNode
23666 QuorumPeerMain
28147 Jps
24472 DFSZKFailoverController
24920 ResourceManager
24265 JournalNode
26301 HMaster
25327 JobHistoryServer
启动master ：start-master.sh
[root@hadoop1 soft]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop1.out
检查
[root@hadoop1 soft]# jps
23968 NameNode
23666 QuorumPeerMain
28293 Master
24472 DFSZKFailoverController
24920 ResourceManager
24265 JournalNode
28363 Jps
26301 HMaster
25327 JobHistoryServer
可以看到master 已经启动
启动slaves ：start-slaves.sh
[root@hadoop1 soft]# start-slaves.sh
hadoop3: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop3.out
hadoop2: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop2.out
检查hadoop2
[root@hadoop2 soft]# jps
15584 Worker
12210 ResourceManager
11731 NameNode
13155 HMaster
13077 HRegionServer
12038 DFSZKFailoverController
15638 Jps
11559 QuorumPeerMain
11931 JournalNode
11820 DataNode
12287 NodeManager
检查hadoop3
[root@hadoop3 soft]# jps
5088 JournalNode
4979 DataNode
5763 HRegionServer
4813 QuorumPeerMain
5245 NodeManager
7390 Worker
7455 Jps
修改SPARK_HOME/sbin 下的start-all.sh 和stop-all.sh 这两个文件的名字
上面是单独启动master 和slaves ，我们也可以使用start-all.sh 和stop-all.sh 来启动会关闭spark 集群。
在使用这两个命令之前，我们可以修改这两个文件名字，如start-spark-all.sh 和stop-spark-all.sh
原因：
如果集群中也配置HADOOP_HOME ，那么在HADOOP_HOME/sbin 目录下也有start-all.sh 和stop-all.sh 这两个文件，当你执行这两个文件，系统不知道是操作hadoop 集群还是spark 集群。修改后就不会冲突了，当然，不修改的话，你需要进入它们的sbin 目录下执行这些文件，这肯定就不会发生冲突了。我们配置SPARK_HOME 主要也是为了执行其他spark 命令方便。
[root@hadoop2 soft]# cd /hadoop/spark/spark-3.0.0-bin-hadoop3.2/sbin/
[root@hadoop2 sbin]# mv start-all.sh start-spark-all.sh
[root@hadoop2 sbin]# mv stop-all.sh stop-spark-all.sh
修改后可以使用下面方法启动关闭spark 集群
[root@hadoop1 sbin]# start-spark-all.sh
starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop1.out
hadoop3: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop3.out
hadoop2: starting org.apache.spark.deploy.worker.Worker, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-hadoop2.out
[root@hadoop1 sbin]# jps
23968 NameNode
29168 Jps
23666 QuorumPeerMain
29107 Master
24472 DFSZKFailoverController
24920 ResourceManager
24265 JournalNode
26301 HMaster
25327 JobHistoryServer
[root@hadoop2 ~]# jps
16352 Worker
16433 Jps
12210 ResourceManager
11731 NameNode
13155 HMaster
13077 HRegionServer
12038 DFSZKFailoverController
11559 QuorumPeerMain
11931 JournalNode
11820 DataNode
12287 NodeManager
[root@hadoop3 sbin]# jps
5088 JournalNode
7936 Jps
4979 DataNode
5763 HRegionServer
7863 Worker
4813 QuorumPeerMain
5245 NodeManager
在hadoop2 上, 启动master ，实现master 的HA
[root@hadoop2 conf]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop2.out
[root@hadoop2 conf]# jps
17584 Jps
12210 ResourceManager
11731 NameNode
13155 HMaster
13077 HRegionServer
17541 Master
12038 DFSZKFailoverController
11559 QuorumPeerMain
11931 JournalNode
11820 DataNode
12287 NodeManager
17439 Worker
3.7 查看spark web
http://192.168.242.81:8888/
http://192.168.242.82:8888/
3.8 测试主备自动切换
在hadoop1 节点中查看并杀掉master 进程
root@hadoop1 conf]# jps
23968 NameNode
23666 QuorumPeerMain
24472 DFSZKFailoverController
24920 ResourceManager
29960 Master
24265 JournalNode
26301 HMaster
30125 Jps
25327 JobHistoryServer
[root@hadoop1 conf]# kill -9 29960
[root@hadoop1 conf]# jps
23968 NameNode
23666 QuorumPeerMain
30147 Jps
24472 DFSZKFailoverController
24920 ResourceManager
24265 JournalNode
26301 HMaster
25327 JobHistoryServer
Hadoop2 自动切换成master 的alive 节点
自动切换后，原来运行的job 应该也不受影响，这里不做演示。
启动hadoop1 的master
[root@hadoop1 conf]# start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /hadoop/spark/spark-3.0.0-bin-hadoop3.2/logs/spark-root-org.apache.spark.deploy.master.Master-1-hadoop1.out
[root@hadoop1 conf]# jps
23968 NameNode
23666 QuorumPeerMain
30213 Master
24472 DFSZKFailoverController
24920 ResourceManager
30248 Jps
24265 JournalNode
26301 HMaster
25327 JobHistoryServer
Hadoop2 自动切换成master 的standby 节点
至此，spark 的HA 集群搭建完成。