主机列表
DC 18.163.111.34 NPS 18.163.35.186 RRAS 18.162.114.236 PC 18.163.117.102
单机环境下RRAS(VPN)服务属性的安全标签页面
将本地用户Administrator的拨入权限设置为允许
在客户机PC上新建拨号VPN连接
查看VPN拨号连接属性配置
修改安全标签页设置
使用RRAS主机用户Administrator进行拨号连接
查看已建立连接的VPN会话信息（本地用户）
在活动目录中新建一个账户并设置拨入的网络接入权限为通过NPS网络策略控制
修改RRAS主机的安全标签页设置
重启RRAS服务
修改客户机PC的VPN连接账户为活动目录中新增用户账户
进行拨号连接
查看已建立连接的VPN会话信息（域用户）
主机列表
DC 18.163.111.34 NPS 18.163.35.186 RRAS 18.162.114.236 PC 18.163.117.102
修改主机IP地址信息为静态IP并指向DNS到AD DC服务器
修改主机名
将NPS主机加入域
重启后使用域管理员账户登录NPS主机
添加角色
启动并配置NPS服务
将NPS注册到活动目录域服务
在活动目录中查看已加入域的计算机RADIUS所属成员组
配置RADIUS客户端
重启NPS服务
禁用SElinux设置
[root@ip-172-31-37-47 ~]# sudo setenforce 0 [root@ip-172-31-37-47 ~]# sudo sed -i 's/^SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
使用Zmodem传输JDK安装文件
[root@ip-172-31-37-47 ~]# yum -y install lrzsz [root@ip-172-31-37-47 ~]# rz rz waiting to receive. Starting zmodem transfer. Press Ctrl+C to cancel. Transferring jdk-8u241-linux-x64.rpm... 100% 174745 KB 417 KB/sec 00:06:59 0 Errors [root@ip-172-31-37-47 ~]#
安装Java运行环境
[root@ip-172-31-37-47 ~]# yum -y install jdk-8u241-linux-x64.rpm [root@ip-172-31-37-47 ~]# java -version java version "1.8.0_241" Java(TM) SE Runtime Environment (build 1.8.0_241-b07) Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode) [root@ip-172-31-37-47 ~]#
添加非特权用户
[root@ip-172-31-37-47 ~]# adduser hadoop [root@ip-172-31-37-47 ~]# passwd hadoop Changing password for user hadoop. New password: Retype new password: passwd: all authentication tokens updated successfully. [root@ip-172-31-37-47 ~]#
切换用户并生成密钥
[root@ip-172-31-37-47 ~]# su - hadoop [hadoop@ip-172-31-37-47 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa Generating public/private rsa key pair. Created directory '/home/hadoop/.ssh'. Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: SHA256:Ji2ZgPoh744id3m6UkIqNzAShPYKLagwVtnulKJWepA hadoop@ip-172-31-37-47.ap-east-1.compute.internal The key's randomart image is: +---[RSA 2048]----+ |o. o | |o. + . | |oo= o . | |XEo+ = + | |OBB + = S | |+O+o.. + | |o.++ . | |ooo o . | |+oooo+ | +----[SHA256]-----+ [hadoop@ip-172-31-37-47 ~]$ cat .ssh/id_rsa.pub >> .ssh/authorized_keys [hadoop@ip-172-31-37-47 ~]$ chmod 0600 .ssh/authorized_keys [hadoop@ip-172-31-37-47 ~]$
下载hadoop最新版本二进制包
[hadoop@ip-172-31-37-47 ~]$ curl -O https://mirrors.ustc.edu.cn/apache/hadoop/common/hadoop-3.1.3/hadoop-3.1.3.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 322M 100 322M 0 0 8877k 0 0:00:37 0:00:37 --:--:-- 16.8M [hadoop@ip-172-31-37-47 ~]$
解压缩并设置软链接
[hadoop@ip-172-31-37-47 ~]$ tar xzf hadoop-3.1.3.tar.gz [hadoop@ip-172-31-37-47 ~]$ ln -s hadoop-3.1.3 hadoop [hadoop@ip-172-31-37-47 ~]$
编辑当前用户环境变量配置
[hadoop@ip-172-31-37-47 ~]$ vi .bashrc export HADOOP_HOME=/home/hadoop/hadoop export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
使环境变量生效
[hadoop@ip-172-31-37-47 ~]$ source .bashrc [hadoop@ip-172-31-37-47 ~]$
设置hadoop配置文件中的JAVA_HOME路径
进入配置文件目录并修改配置文件
[hadoop@ip-172-31-37-47 ~]$ cd hadoop/etc/hadoop/ [hadoop@ip-172-31-37-47 hadoop]$ ls capacity-scheduler.xml httpfs-log4j.properties mapred-site.xml configuration.xsl httpfs-signature.secret shellprofile.d container-executor.cfg httpfs-site.xml ssl-client.xml.example core-site.xml kms-acls.xml ssl-server.xml.example hadoop-env.cmd kms-env.sh user_ec_policies.xml.template hadoop-env.sh kms-log4j.properties workers hadoop-metrics2.properties kms-site.xml yarn-env.cmd hadoop-policy.xml log4j.properties yarn-env.sh hadoop-user-functions.sh.example mapred-env.cmd yarnservice-log4j.properties hdfs-site.xml mapred-env.sh yarn-site.xml httpfs-env.sh mapred-queues.xml.template [hadoop@ip-172-31-37-47 hadoop]$ [hadoop@ip-172-31-37-47 hadoop]$ vi hadoop-env.sh # The java implementation to use. By default, this environment # variable is REQUIRED on ALL platforms except OS X! # export JAVA_HOME= export JAVA_HOME=/usr/java/jdk1.8.0_241-amd64 [hadoop@ip-172-31-37-47 hadoop]$ vi core-site.xml <configuration> <property> <name>fs.default.name</name> <value>hdfs://localhost:9000</value> </property> </configuration> [hadoop@ip-172-31-37-47 hadoop]$ vi hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.name.dir</name> <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value> </property> <property> <name>dfs.data.dir</name> <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value> </property> </configuration> [hadoop@ip-172-31-37-47 hadoop]$ vi yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
对Namenode进行格式化
[hadoop@ip-172-31-37-47 ~]$ hdfs namenode -format WARNING: /home/hadoop/hadoop/logs does not exist. Creating. 2020-03-06 03:49:28,538 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = ip-172-31-37-47.ap-east-1.compute.internal/172.31.37.47 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 3.1.3 STARTUP_MSG: classpath = /home/hadoop/hadoop/etc/hadoop:/home/hadoop/hadoop/share/hadoop/common/lib/accessors-smart-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/animal-sniffer-annotations-1.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/asm-5.0.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/avro-1.7.7.jar:/home/hadoop/hadoop/share/hadoop/common/lib/checker-qual-2.5.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-codec-1.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-compress-1.18.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-io-2.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-lang3-3.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/commons-net-3.6.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-client-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-framework-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/error_prone_annotations-2.2.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/failureaccess-1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-annotations-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/hadoop-auth-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/hadoop/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/j2objc-annotations-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-databind-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-core-1.19.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-json-1.19.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-server-1.19.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jersey-servlet-1.19.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/home/hadoop/hadoop/share/hadoop/common/lib/json-smart-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-client-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-common-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-core-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-server-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerb-util-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerby-config-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/hadoop/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/common/lib/netty-3.10.5.Final.jar:/home/hadoop/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/common/lib/re2j-1.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/home/hadoop/hadoop/share/hadoop/common/lib/snappy-java-1.0.5.jar:/home/hadoop/hadoop/share/hadoop/common/lib/stax2-api-3.1.4.jar:/home/hadoop/hadoop/share/hadoop/common/lib/token-provider-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/home/hadoop/hadoop/share/hadoop/common/lib/zookeeper-3.4.13.jar:/home/hadoop/hadoop/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/home/hadoop/hadoop/share/hadoop/common/lib/metrics-core-3.2.4.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-common-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-nfs-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/common/hadoop-kms-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/okio-1.6.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/hadoop-auth-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/json-smart-2.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/asm-5.0.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/guava-27.0-jre.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/failureaccess-1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/checker-qual-2.5.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/error_prone_annotations-2.2.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/j2objc-annotations-1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/animal-sniffer-annotations-1.17.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-io-2.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/hadoop-annotations-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-net-3.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jettison-1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/avro-1.7.7.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/paranamer-2.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/re2j-1.1.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/gson-2.2.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/home/hadoop/hadoop/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3-tests.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn:/home/hadoop/hadoop/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/fst-2.50.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-4.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/java-util-1.9.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-client-1.19.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/json-io-2.5.1.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/objenesis-1.0.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/home/hadoop/hadoop/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-api-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-router-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.1.3.jar:/home/hadoop/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.1.3.jar STARTUP_MSG: build = https://gitbox.apache.org/repos/asf/hadoop.git -r ba631c436b806728f8ec2f54ab1e289526c90579; compiled by 'ztang' on 2019-09-12T02:47Z STARTUP_MSG: java = 1.8.0_241 ************************************************************/ 2020-03-06 03:49:28,545 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2020-03-06 03:49:28,629 INFO namenode.NameNode: createNameNode [-format] 2020-03-06 03:49:28,725 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Formatting using clusterid: CID-e8b28488-ea5a-40b3-8ba7-f6e4c14d0e0f 2020-03-06 03:49:29,071 INFO namenode.FSEditLog: Edit logging is async:true 2020-03-06 03:49:29,082 INFO namenode.FSNamesystem: KeyProvider: null 2020-03-06 03:49:29,083 INFO namenode.FSNamesystem: fsLock is fair: true 2020-03-06 03:49:29,083 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false 2020-03-06 03:49:29,086 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 2020-03-06 03:49:29,087 INFO namenode.FSNamesystem: supergroup = supergroup 2020-03-06 03:49:29,087 INFO namenode.FSNamesystem: isPermissionEnabled = true 2020-03-06 03:49:29,087 INFO namenode.FSNamesystem: HA Enabled: false 2020-03-06 03:49:29,120 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2020-03-06 03:49:29,128 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000 2020-03-06 03:49:29,128 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 2020-03-06 03:49:29,131 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 2020-03-06 03:49:29,134 INFO blockmanagement.BlockManager: The block deletion will start around 2020 Mar 06 03:49:29 2020-03-06 03:49:29,135 INFO util.GSet: Computing capacity for map BlocksMap 2020-03-06 03:49:29,135 INFO util.GSet: VM type = 64-bit 2020-03-06 03:49:29,136 INFO util.GSet: 2.0% max memory 3.4 GB = 69.8 MB 2020-03-06 03:49:29,136 INFO util.GSet: capacity = 2^23 = 8388608 entries 2020-03-06 03:49:29,151 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false 2020-03-06 03:49:29,156 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS 2020-03-06 03:49:29,156 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 2020-03-06 03:49:29,156 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0 2020-03-06 03:49:29,156 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: defaultReplication = 1 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: maxReplication = 512 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: minReplication = 1 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: encryptDataTransfer = false 2020-03-06 03:49:29,157 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 2020-03-06 03:49:29,199 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215 2020-03-06 03:49:29,212 INFO util.GSet: Computing capacity for map INodeMap 2020-03-06 03:49:29,212 INFO util.GSet: VM type = 64-bit 2020-03-06 03:49:29,212 INFO util.GSet: 1.0% max memory 3.4 GB = 34.9 MB 2020-03-06 03:49:29,212 INFO util.GSet: capacity = 2^22 = 4194304 entries 2020-03-06 03:49:29,214 INFO namenode.FSDirectory: ACLs enabled? false 2020-03-06 03:49:29,214 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true 2020-03-06 03:49:29,214 INFO namenode.FSDirectory: XAttrs enabled? true 2020-03-06 03:49:29,214 INFO namenode.NameNode: Caching file names occurring more than 10 times 2020-03-06 03:49:29,218 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536 2020-03-06 03:49:29,219 INFO snapshot.SnapshotManager: SkipList is disabled 2020-03-06 03:49:29,222 INFO util.GSet: Computing capacity for map cachedBlocks 2020-03-06 03:49:29,222 INFO util.GSet: VM type = 64-bit 2020-03-06 03:49:29,222 INFO util.GSet: 0.25% max memory 3.4 GB = 8.7 MB 2020-03-06 03:49:29,222 INFO util.GSet: capacity = 2^20 = 1048576 entries 2020-03-06 03:49:29,253 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10 2020-03-06 03:49:29,253 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10 2020-03-06 03:49:29,253 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25 2020-03-06 03:49:29,257 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 2020-03-06 03:49:29,257 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 2020-03-06 03:49:29,258 INFO util.GSet: Computing capacity for map NameNodeRetryCache 2020-03-06 03:49:29,258 INFO util.GSet: VM type = 64-bit 2020-03-06 03:49:29,259 INFO util.GSet: 0.029999999329447746% max memory 3.4 GB = 1.0 MB 2020-03-06 03:49:29,259 INFO util.GSet: capacity = 2^17 = 131072 entries 2020-03-06 03:49:29,279 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1445179826-172.31.37.47-1583466569274 2020-03-06 03:49:29,289 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted. 2020-03-06 03:49:29,310 INFO namenode.FSImageFormatProtobuf: Saving image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 using no compression 2020-03-06 03:49:29,386 INFO namenode.FSImageFormatProtobuf: Image file /home/hadoop/hadoopdata/hdfs/namenode/current/fsimage.ckpt_0000000000000000000 of size 393 bytes saved in 0 seconds . 2020-03-06 03:49:29,395 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 2020-03-06 03:49:29,399 INFO namenode.FSImage: FSImageSaver clean checkpoint: txid = 0 when meet shutdown. 2020-03-06 03:49:29,399 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-37-47.ap-east-1.compute.internal/172.31.37.47 ************************************************************/ [hadoop@ip-172-31-37-47 ~]$
启动hadoop集群
[hadoop@ip-172-31-37-47 sbin]$ ./start-dfs.sh Starting namenodes on [localhost] Starting datanodes Starting secondary namenodes [ip-172-31-37-47.ap-east-1.compute.internal] ip-172-31-37-47.ap-east-1.compute.internal: Warning: Permanently added 'ip-172-31-37-47.ap-east-1.compute.internal,172.31.37.47' (ECDSA) to the list of known hosts. 2020-03-06 03:50:47,021 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [hadoop@ip-172-31-37-47 sbin]$ [hadoop@ip-172-31-37-47 sbin]$ ./start-yarn.sh Starting resourcemanager Starting nodemanagers [hadoop@ip-172-31-37-47 sbin]$
查看端口监听
[hadoop@ip-172-31-37-47 ~]$ netstat -lntp (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:8040 0.0.0.0:* LISTEN 15150/java tcp 0 0 0.0.0.0:9864 0.0.0.0:* LISTEN 14583/java tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 14436/java tcp 0 0 0.0.0.0:8042 0.0.0.0:* LISTEN 15150/java tcp 0 0 0.0.0.0:9866 0.0.0.0:* LISTEN 14583/java tcp 0 0 0.0.0.0:41963 0.0.0.0:* LISTEN 15150/java tcp 0 0 0.0.0.0:9867 0.0.0.0:* LISTEN 14583/java tcp 0 0 0.0.0.0:9868 0.0.0.0:* LISTEN 14777/java tcp 0 0 0.0.0.0:9870 0.0.0.0:* LISTEN 14436/java tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:35031 0.0.0.0:* LISTEN 14583/java tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 15026/java tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:13562 0.0.0.0:* LISTEN 15150/java tcp 0 0 0.0.0.0:8030 0.0.0.0:* LISTEN 15026/java tcp 0 0 0.0.0.0:8031 0.0.0.0:* LISTEN 15026/java tcp 0 0 0.0.0.0:8032 0.0.0.0:* LISTEN 15026/java tcp 0 0 0.0.0.0:8033 0.0.0.0:* LISTEN 15026/java tcp6 0 0 :::111 :::* LISTEN - tcp6 0 0 :::22 :::* LISTEN - tcp6 0 0 ::1:25 :::* LISTEN - [hadoop@ip-172-31-37-47 ~]$
在浏览器中访问Hadoop服务Web控制台
查看Namenode状态
查看集群状态
查看节点详情
