Docker搭建Spark集群,你学会了吗?

docker搭建spark集群

有个小技巧:先配置好一个,在(宿主机上)复制scp -r拷贝Spark到其他Slaves。

1.安装配置基础Spark

【在test-cluster-hap-master-01虚拟主机上】

将已下载好的Spark压缩包(spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt.tgz)通过工具【XFtp】拷贝到虚拟主机的opt目录下:

2.通过脚本挂起镜像

cd   /opt/script/setup/spark

test-cluster-spk-master-01

#!/bin/bash 
#编写作者:千羽的编程时光

cname="test-cluster-spk-master-01"

#port1="8080"
#port2="7077"
log="/opt/data/"${cname}
images="10.249.0.137:80/base/jdk-1.8:20210202"

mkdir -p ${log}
mkdir ${log}/logs
mkdir ${log}/work
mkdir ${log}/data
mkdir ${log}/jars

# docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always 
docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always 
-v ${log}/logs:/usr/local/spark-3.1.1/logs 
-v ${log}/work:/usr/local/spark-3.1.1/work 
-v ${log}/jars:/usr/local/spark-3.1.1/jars 
-v ${log}/data:/opt/data 
${images} 
/usr/sbin/init

test-cluster-spk-master-02

#!/bin/bash 
cname="test-cluster-spk-master-02"

#port1="8080"
#port2="7077"
log="/opt/data/"${cname}
images="10.249.0.137:80/base/jdk-1.8:20210202"

mkdir -p ${log}
mkdir ${log}/logs
mkdir ${log}/work
mkdir ${log}/data
mkdir ${log}/jars

#docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always 
docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always 
-v ${log}/logs:/usr/local/spark-3.1.1/logs 
-v ${log}/work:/usr/local/spark-3.1.1/work 
-v ${log}/jars:/usr/local/spark-3.1.1/jars 
-v ${log}/data:/opt/data 
${images} 
/usr/sbin/init

test-cluster-spk-slave-01

#!/bin/bash 
cname="test-cluster-spk-slave-01"

#port1="8080"
#port2="7077"
log="/opt/data/"${cname}
images="10.249.0.137:80/base/jdk-1.8:20210202"

mkdir -p ${log}
mkdir ${log}/logs
mkdir ${log}/work
mkdir ${log}/data
mkdir ${log}/jars

#docker run -d --net=overlay-net --ip ${ip} -p ${port1}:${port1} -p ${port2}:${port2} --name ${cname} --hostname ${cname} --privileged=true --restart=always 
docker run -d --net=host --name ${cname} --hostname ${cname} --privileged=true --restart=always 
-v ${log}/logs:/usr/local/spark-3.1.1/logs 
-v ${log}/work:/usr/local/spark-3.1.1/work 
-v ${log}/jars:/usr/local/spark-3.1.1/jars 
-v ${log}/data:/opt/data 
${images} 
/usr/sbin/init
[root@zookeeper-03-test spark]# ll
总用量 4
-rw-r--r--. 1 root root 1166 7月  28 17:44 install.sh
[root@zookeeper-03-test spark]# chmod +x install.sh 
[root@zookeeper-03-test spark]# ll
总用量 4
-rwxr-xr-x. 1 root root 1166 7月  28 17:44 install.sh
[root@zookeeper-03-test spark]#

3.上传spark安装包

在容器映射目录下 :/opt/data/test-cluster-spk-slave-01/data

[root@hadoop-01 data]# pwd
/opt/data

用Xftp上传包

Docker搭建Spark集群,你学会了吗?

这里需要上传两个,使用的是spark-3.1.1-bin-without-hadoop.tgz

但是需要将spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt下的jars包移到/usr/local/spark-3.1.1/jars下

4.解压安装包

mkdir -p /usr/local/spark-3.1.1
cd /opt/data
tar -zxvf spark-3.1.1-bin-without-hadoop.tgz -C /usr/local/spark-3.1.1

Docker搭建Spark集群,你学会了吗?

Docker搭建Spark集群,你学会了吗?


Docker搭建Spark集群,你学会了吗?

编辑全局变量

vim /etc/profile

增加以下全局变量

export SPARK_HOME=/usr/local/spark-3.1.1   
export PATH=$PATH:$SPARK_HOME/bin

即时生效

source /etc/profile

5.配置spark-env.sh

cd /usr/local/spark-3.1.1/conf
cp spark-env.sh.template spark-env.sh
vim spark-env.sh
export SPARK_MASTER_IP=test-cluster-spk-master-01
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=800m
#export SPARK_DRIVER_MEMORY=4g
export SPARK_EXECUTOR_INSTANCES=2
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_LOCAL_DIRS=/home/hadoop/tmp/spark/tmp

#定时清理worker文件 一天一次
export SPARK_WORKER_OPTS="  
-Dspark.worker.cleanup.enabled=true  
-Dspark.worker.cleanup.interval=86400 
-Dspark.worker.cleanup.appDataTtl=86400"

export JAVA_HOME=/usr/local/jdk1.8
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SCALA_HOME=/usr/local/scala
export PATH=${SCALA_HOME}/bin:$PATH
export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=zookeeper-01-test:2181,zookeeper-02-test:2181,zookeeper-03-test:2181 -Dspark.deploy.zookeeper.dir=/usr/local/spark"

(4)配置workers

cp workers.template workers
vim workers
# 添加
test-cluster-spk-slave-001

Docker搭建Spark集群,你学会了吗?

6.配置log4j.properties

cp log4j.properties.template log4j.properties
vim log4j.properties

log4j.rootCategory=WARN, console

Docker搭建Spark集群,你学会了吗?

7.复制到其他slave

(宿主机上)复制scp -r拷贝Spark到其他Slaves节点:

scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-001-spark-dev:/usr/local/spark/

scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-002-spark-dev:/usr/local/spark/

scp -r /usr/local/spark/spark-2.1.0-bin-hadoop2.7 root@slave-003-spark-dev:/usr/local/spark/

如执行命令出现出现问题时,请现在相应的Slave节点执行mkdir -p /usr/local/spark

复制到master-02时,使用start-mater.sh启动master-02

8.启动spark

  1. 先启动两个master,然后启动slave节点
[root@test-cluster-spk-master-01 sbin]# ./start-master.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-3.1.1/logs/spark-root-org.apache.spark.deploy.master.Master-1-test-cluster-spk-master-01.out
[root@test-cluster-spk-master-01 sbin]# jps
548 Jps
492 Master
[root@test-cluster-spk-master-01 sbin]# pwd
/usr/local/spark-3.1.1/sbin
[root@test-cluster-spk-master-01 sbin]#
  1. 主节点2启动完成
[root@test-cluster-spk-master-02 sbin]# ./start-master.sh 
starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark-3.1.1/logs/spark-root-org.apache.spark.deploy.master.Master-1-test-cluster-spk-master-02.out
[root@test-cluster-spk-master-02 sbin]# pwd
/usr/local/spark-3.1.1/sbin
[root@test-cluster-spk-master-02 sbin]# jps
274 Jps
218 Master
[root@test-cluster-spk-master-02 sbin]#
  1. 从节点启动完成
/usr/local/spark-3.1.1/sbin/start-slave.sh test-cluster-hap-slave-001 test-cluster-hap-master-02:7077,test-cluster-hap-master-02:7077
  1. 主节点1启动完成

9.验证

原本是访问http://10.8.46.35:8080 就可,但是我这里在配置镜像的时候,多了8080,导致这里访问不了。看日志可以知道,已经走向8081

所以http://10.8.46.35:8081/即可

主节点1

停掉主节点

Docker搭建Spark集群,你学会了吗?

Docker搭建Spark集群,你学会了吗?

Docker搭建Spark集群,你学会了吗?

主节点2

从节点成为ALIVE

Docker搭建Spark集群,你学会了吗?

Docker搭建Spark集群,你学会了吗?

从节点1

从节点1

Docker搭建Spark集群,你学会了吗?

Docker搭建Spark集群,你学会了吗?


10.遇到的坑

包不兼容

这里遇到了许多问题,第一个是包不兼容,导致搭建两次失败

Docker搭建Spark集群,你学会了吗?

然后换了官方的包spark-3.1.1-bin-without-hadoop,启动还是有问题。

最后通过替换jars才成功。(使用spark-3.1.1-bin-hadoop-3.2.2-lbx-jszt下的jars)

ctrl + p + q    从容器退出到宿主机

done~

文章版权声明

 1 原创文章作者:39791,如若转载,请注明出处: https://www.52hwl.com/30484.html

 2 温馨提示:软件侵权请联系469472785#qq.com(三天内删除相关链接)资源失效请留言反馈

 3 下载提示:如遇蓝奏云无法访问,请修改lanzous(把s修改成x)

 免责声明:本站为个人博客,所有软件信息均来自网络 修改版软件,加群广告提示为修改者自留,非本站信息,注意鉴别

(0)
打赏 微信扫一扫 微信扫一扫 支付宝扫一扫 支付宝扫一扫
上一篇 2023年7月14日 上午12:00
下一篇 2023年7月15日