一、环境介绍
1.四台CentOS7 Linux虚拟机机器分布情况:
192.168.0.1 安装NameNode,ResourceManager和SecondaryNameNode
192.168.0.2 安装NodeManager和DataNode
192.168.0.3 安装NodeManager和DataNode
192.168.0.4 安装NodeManager和DataNode
2.配置DNS(每个节点)
编辑配置文件,添加主节点和从节点的映射关系。
#vi /etc/hosts
192.168.0.1 mdw2 hadoop01
192.168.0.2 mdw3 hadoop02
192.168.0.3 mdw4 hadoop03
192.168.0.4 mdw5 hadoop04
3. 关闭防火墙(每个节点)
# systemctl stop firewalld
#关闭开机自启动
# systemctl disable firewalld
4. 配置免密码登录
有关【配置免密码登录方法】,请参考
https://www.cnblogs.com/shireenlee4testing/p/10366061.html
5. 配置Java环境(每个节点)
有关【配置java环境方法】,请参考
https://www.cnblogs.com/shireenlee4testing/p/10368961.html
二、搭建Hadoop完全分布式集群
1. 下载Hadoop安装包,解压,配置Hadoop环境变量
# wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz
#解压到/opt目录
# tar -zxvf hadoop-3.2.0.tar.gz
#链接/opt/hadoop-3.2.0到/opt/hadoop,方便后续配置
#ln -s hadoop-3.2.0 hadoop
#配置Hadoop环境变量和java环境变量
# vi /etc/profile
Hadoop
export HADOOP_HOME=/opt/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin export HADOOP_CONF_DIR=${HADOOP_HOME}/etc/hadoop
#jdk
export JAVA_HOME=/opt/jdk
export PATH=$PATH:$JAVA_HOME/bin
2. 配置Hadoop环境脚本文件中的JAVA_HOME参数
#进入Hadoop安装目录下的etc/hadoop目录
# cd /opt/hadoop/etc/hadoop
#分别在hadoop-env.sh、mapred-env.sh和yarn-env.sh 文件中添加或修改如下参数:
# vi hadoop-env.sh
............................................................
............................................................
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME=/opt/jdk
# vi mapred-env.sh
............................................................
............................................................
# Specify the log4j settings for the JobHistoryServer
# Java property: hadoop.root.logger
#export HADOOP_JHS_LOGGER=INFO,RFA
export JAVA_HOME=/opt/jdk
# vi mapred-env.sh
............................................................
............................................................
# Specify the log4j settings for the JobHistoryServer
# Java property: hadoop.root.logger
#export HADOOP_JHS_LOGGER=INFO,RFA
export JAVA_HOME=/opt/jdk
# vi yarn-env.sh
............................................................
............................................................
# YARN Services parameters
###
# Directory containing service examples
# export YARN_SERVICE_EXAMPLES_DIR = $HADOOP_YARN_HOME/share/hadoop/yarn/yarn-service-examples
# export YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE=true
export JAVA_HOME=/opt/jdk
#验证Hadoop配置是否生效
# hadoop version
Hadoop 3.2.0
Source code repository https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf
Compiled by sunilg on 2019-01-08T06:08Z
Compiled with protoc 2.5.0
From source with checksum d3f0795ed0d9dc378e2c785d3668f39
This command was run using /opt/hadoop-3.2.0/share/hadoop/common/hadoop-common-3.2.0.jar
3. 修改Hadoop配置文件
Hadoop安装目录下的etc/hadoop目录中,需修改core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml、workers文件,根据实际情况修改配置信息。
# cat /opt/hadoop/etc/hadoop/core-site.xml
# cat /opt/hadoop/etc/hadoop/hdfs-site.xml
# cat /opt/hadoop/etc/hadoop/mapred-site.xml
# cat /opt/hadoop/etc/hadoop/yarn-site.xml
# cat /opt/hadoop/etc/hadoop/workers
hadoop02
hadoop03
hadoop04
4. 配置启动脚本,添加HDFS和Yarn权限
添加HDFS权限:编辑如下脚本,在第二行空白位置添加HDFS权限
# vi /opt/hadoop/sbin/start-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
# vi /opt/hadoop/sbin/stop-dfs.sh
HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=root
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
添加Yarn权限:编辑如下脚本,在第二行空白位置添加Yarn权限
# vi /opt/hadoop/sbin/start-yarn.sh
YARN_RESOURCEMANAGER_USER=root
HDFS_DATANODE_SECURE_USER=root
YARN_NODEMANAGER_USER=root
# vi /opt/hadoop/sbin/stop-yarn.sh
YARN_RESOURCEMANAGER_USER=root
HDFS_DATANODE_SECURE_USER=root
YARN_NODEMANAGER_USER=root
注意:若不添加上述权限,则会报错:缺少用户权限定义所致。
5. 将配置好的文件夹拷贝到其他从节点
# scp -r /opt/hadoop-3.2.0 root@hadoop02:/opt/
# scp -r /opt/hadoop-3.2.0 root@hadoop03:/opt/
# scp -r /opt/hadoop-3.2.0 root@hadoop04:/opt/
# scp -r /opt/hadoop root@hadoop02:/opt/
# scp -r /opt/hadoop root@hadoop03:/opt/
# scp -r /opt/hadoop root@hadoop04:/opt/
6. 初始化 & 启动
#格式化
[root@hadoop01 hadoop-3.2.0]# /opt/hadoop/bin/hdfs namenode -format
#启动
[root@hadoop01 hadoop-3.2.0]# /opt/hadoop/sbin/start-all.sh
Starting namenodes on [hadoop01]
上一次登录:一 10月 12 16:22:06 CST 2020pts/1 上
Starting datanodes
上一次登录:一 10月 12 16:22:32 CST 2020pts/1 上
Starting secondary namenodes [mdw2]
上一次登录:一 10月 12 16:22:34 CST 2020pts/1 上
Starting resourcemanager
上一次登录:一 10月 12 16:22:40 CST 2020pts/1 上
Starting nodemanagers
上一次登录:一 10月 12 16:22:47 CST 2020pts/1 上
7. 验证Hadoop启动成功
#主节点
[root@mdw2 ~]# jps
5089 NameNode
5625 ResourceManager
99770 Jps
5372 SecondaryNameNode
#从节点
# jps
56978 NodeManager
80172 Jps
56862 DataNode
查看Hadoop集群状态
[root@mdw2 ~]# hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.
Configured Capacity: 160982630400 (149.93 GB)
Present Capacity: 131017445376 (122.02 GB)
DFS Remaining: 131017408512 (122.02 GB)
DFS Used: 36864 (36 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Low redundancy blocks with highest priority to recover: 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (3):
Name: 192.168.0.2:9866 (mdw3)
Hostname: mdw3
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 10945437696 (10.19 GB)
DFS Remaining: 42715426816 (39.78 GB)
DFS Used%: 0.00%
DFS Remaining%: 79.60%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Oct 14 13:55:20 CST 2020
Last Block Report: Wed Oct 14 11:53:54 CST 2020
Num of Blocks: 0
Name: 192.168.0.3:9866 (mdw4)
Hostname: mdw4
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 10945388544 (10.19 GB)
DFS Remaining: 42715475968 (39.78 GB)
DFS Used%: 0.00%
DFS Remaining%: 79.60%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Oct 14 13:55:21 CST 2020
Last Block Report: Wed Oct 14 12:57:21 CST 2020
Num of Blocks: 0
Name: 192.168.0.4:9866 (mdw5)
Hostname: mdw5
Decommission Status : Normal
Configured Capacity: 53660876800 (49.98 GB)
DFS Used: 12288 (12 KB)
Non DFS Used: 8074358784 (7.52 GB)
DFS Remaining: 45586505728 (42.46 GB)
DFS Used%: 0.00%
DFS Remaining%: 84.95%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Oct 14 13:55:20 CST 2020
Last Block Report: Wed Oct 14 12:17:55 CST 2020
Num of Blocks: 0
单独启动resourcemanager:
[root@mdw2 hadoop]# yarn-daemon.sh start resourcemanager
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
[root@mdw2 hadoop]# jps
35411 NameNode
35691 SecondaryNameNode
38558 Jps
38319 ResourceManager
8. Web端口访问
http://192.168.0.1:50070/
http://192.168.0.1:8088/
从节点NodeManager进程启动不起来的解决方案:
NodeManager进程报错信息如下:
INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
原因:Hadoop集群yarn-site.xml配置错误所致:
默认情况下yarn ResourceManager 相关服务IP地址指向的是0.0.0.0。
而在服务器中,0.0.0.0指的是本机网络地址,那么NodeManager就会在本机找ResourceManager相关服务,而slave节点上并没有这些服务,这些服务在ResourceManager Master主节点上。
所以针对Hadoop集群配置yare-site.xml某些配置项不能使用默认配置。
解决方法:
修改hadoop集群所有节点上yarn-site.xml配置文件,在该文件中配置ResourceManager Master主节点所在地址即可解决问题。详细配置信息如下:
# vi /opt/hadoop/etc/hadoop/yarn-site.xml,在
NodeManager进程正常启动后的日志如下:
2020-10-13 14:15:53,762 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@14b030a0{/static,jar:file:/opt/hadoop-3.2.0/share/hadoop/yarn/hadoop-yarn-common-3.2.0.jar!/webapps/static,AVAILABLE}
2020-10-13 14:15:55,165 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@2b5183ec{/,file:///tmp/jetty-0.0.0.0-8042-node-_-any-5774776794028847658.dir/webapp/,AVAILABLE}{/node}
2020-10-13 14:15:55,186 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@5eb2172{HTTP/1.1,[http/1.1]}{0.0.0.0:8042}
2020-10-13 14:15:55,186 INFO org.eclipse.jetty.server.Server: Started @5011ms
2020-10-13 14:15:55,186 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app node started at 8042
2020-10-13 14:15:55,210 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Node ID assigned is : mdw3:24558
2020-10-13 14:15:55,218 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.0.1:8031
2020-10-13 14:15:55,223 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor
2020-10-13 14:15:55,323 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
2020-10-13 14:15:55,349 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
2020-10-13 14:15:55,520 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Rolling master-key for container-tokens, got key with id -955208939
2020-10-13 14:15:55,521 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: Rolling master-key for container-tokens, got key with id -1467324462
2020-10-13 14:15:55,522 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as mdw3:24558 with total resource of