Environment to prepare

The server memory CPU system Network IP
k8s-master 4G 2 the nuclear Centos 7.6 192.168.0.6
k8s-node1 4G 2 the nuclear Centos 7.6 192.168.0.47
k8s-node2 4G 2 the nuclear Centos 7.6 192.168.0.154

Configure hosts

192.168.0.6    	k8s-master
192.168.0.47	k8s-node1
192.168.0.154	k8s-node2
Copy the code

Configure no-password login

1. Generate a secret key, ssh-keygen -t rsa. 2 k8s-node1 ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-node2 3. Test Login without password SSH k8s-master SSH k8s-node1 SSH k8s-node2Copy the code

Install the JDK

1. Check the available JDK version yum Search JDK 2. Install JDK yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel 3. Uninstall JDK yum remove java-1.8.0-openjdk java-1.8.0-openjdk -devel-yCopy the code

Install hadoop

1. Visit the Hadoop official website

https://hadoop.apache.org/releases.html 
Copy the code

2. Select hadoop3.2.1 and Binary download

Run the mkdir /usr/local/bigdata CD /usr/local/bigdata wget command on k8s-master https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz tar - ZXVF hadoop - 3.2.1. Tar. Gz mv Hadoop-3.2.1 Hadoop vi readme. Md HaddOOP Version :3.2.1 Environment variable configuration: XML. Modify hdFs-site. XML. Modify mapred-site. XMLCopy the code

3. Configure environment variables

Vi/etc/profile quick command G o shift + insert export HADOOP_HOME = / usr/local/bigdata/hadoop export PATH = $PATH: $HADOOP_HOME/bin Export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export JAVA_HOME=/usr/lib/ JVM /java-1.8.0-openjdk export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar source /etc/profile Checks whether environment variables are configured successfully $HADOOP_HOMECopy the code

4. Modify the configuration file

core-site.xml

cd /usr/local/bigdata/hadoop/etc/hadoop
vi core-site.xml
Copy the code
<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/usr/local/bigdata/hadoop/tmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://k8s-master:9000</value>
    </property>
</configuration>
                                                                                                                   
Copy the code

hdfs-site.xml

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.name.dir</name>
        <value>/usr/local/bigdata/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.data.dir</name>
        <value>/usr/local/bigdata/hadoop/hdfs/data</value>
    </property>
</configuration>
Copy the code

mapred-site.xml

<configuration>
  <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_HOME/share/hadoop/mapreduce/*:$HADOOP_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>
Copy the code

yarn-site.xml

<configuration> <! -- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>k8s-master</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP _HOME</value> </property> </configuration>Copy the code

workers

vi workers
k8s-master
k8s-node1
k8s-node2                   
Copy the code

For details about the Configuration, see Configuration

Hadoop.apache.org/docs/stable…

5. To configure the namenode

yum install zip zip -r hadoop.zip hadoop/ scp hadoop.zip root@k8s-node1:/usr/local/hadoop/ scp hadoop.zip root@k8s-node2: /usr/local/hadoop/Repeat the JDK configuration on k8s-node1 k8s-node2, step CD /usr/local/hadoop unzip hadoop.zipCopy the code

6. Start the cluster

Format HDFS bin/hadoop namenode-format Start the cluster sbin/start-all.shCopy the code

7. An error is reported when the root user is started

ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as root ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation. Starting secondary namenodes [k8s-master] ERROR: Attempting to operate on hdfs secondarynamenode as root ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation. Starting resourcemanager ERROR: Attempting to operate on yarn resourcemanager as root ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation. Starting nodemanagers ERROR: Attempting to operate on yarn nodemanager as root ERROR: but there is no YARN_NODEMANAGER_USER defined. Aborting operation. Solution: Sh stop-dfs.sh Add HDFS_DATANODE_USER=root HADOOP_SECURE_DN_USER= HDFS HDFS_NAMENODE_USER=root HDFS_SECONDARYNAMENODE_USER=root sbin/start-yarn.sh stop-yarn.sh Add YARN_RESOURCEMANAGER_USER=root HADOOP_SECURE_DN_USER=yarn YARN_NODEMANAGER_USER=rootCopy the code

8.k8s-master: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

SSH /id_rsa.pub k8s-master for login without password :ssh-copy-id -i ~/. SSH /id_rsa.pub k8s-masterCopy the code

9.JAVA_HOME is not set

k8s-master: ERROR: JAVA_HOME is not set and could not be found.
k8s-node2: ERROR: JAVA_HOME is not set and could not be found.
k8s-node1: ERROR: JAVA_HOME is not set and could not be found
Copy the code
Three nodes perform CD/usr/local/bigdata/hadoop/etc/hadoop vi hadoop - env. Sh export JAVA_HOME = / usr/lib/JVM/Java -- 1.8.0 comes with itsCopy the code

10. Resolve the error and start the cluster again

Sbin /start-all.sh The following message is displayed: WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER. Starting namenodes on [k8s-master] Last login: Wed Jan 20 23:34:59 CST 2021 on pts/0 Starting datanodes Last login: Wed Jan 20 23:40:33 CST 2021 on pts/0 k8s-node1: WARNING: /usr/local/bigdata/hadoop/logs does not exist. Creating. k8s-node2: WARNING: /usr/local/bigdata/hadoop/logs does not exist. Creating. Starting secondary namenodes [k8s-master] Last login: Wed Jan 20 23:40:35 CST 2021 on pts/0 Starting resourcemanager Last login: Wed Jan 20 23:40:38 CST 2021 on pts/0 resourcemanager is running as process 21204. Stop it first. Starting nodemanagers Last login: Wed Jan 20 23:40:44 CST 2021 on PTS / 0 can view/usr/local/bigdata/hadoop/logs folder log SSH k8s rac-node1 tail -f Hadoop-root -datanode-k8s-node1.log Error: Retrying connect to server: K8s -master/192.168.0.6:9000 Check whether processes exist on the K8s -master 9000 port. Lsof -i:9000 Java 23342 root 277U IPv4 57218 0T0 TCP localhost:cslistener (LISTEN) java 23342 root 287u IPv4 57703 0t0 TCP localhost:cslistener->localhost:40644 (ESTABLISHED) Java 23518 root 315U IPv4 58624 0t0 TCP localhost:40644->localhost: VI exists in the CSListener (ESTABLISHED) process # 127.0.0.1k8s -master # 127.0.0.1k8s -node1 # 127.0.0.1k8s -node2Copy the code

11. View cluster status hadoop dfsadmin-report

WARNING: Use of this script to execute dfsadmin is deprecated. WARNING: Attempting to execute replacement "hdfs dfsadmin" instead. Configured Capacity: Present Capacity: 72120131584 (67.17 GB) DFS Remaining: 72120082432 (67.17 GB) DFS Used: DFS Used%: 0.00% Replicated Blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 Erasure Coded Block Groups: Low redundancy block groups: 0 Block groups with corrupt internal blocks: 0 Missing block groups: 0 Low redundancy blocks with highest priority to recover: 0 Pending deletion blocks: 0 -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Live datanodes (2) : Name: 192.168.0.154:9866 (k8s - 2) the Hostname: K8s-node2 Decommission Status: Normal Configured Capacity: 42140401664 DFS Used: 24576 (24 KB) Non DFS Used: 4114890752 (3.83 GB) DFS Remaining: 35861278720 (33.40 GB) DFS Used%: 0.00% DFS Remaining%: 85.10% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last Contact: Wed Jan 20 23:56:42 CST 2021 Last Block Report: Wed Jan 20 23:53:06 CST 2021 Num of Blocks: 0 Name: 192.168.0.77:9866 (k8s-node1) Hostname: K8s-node1 Decommission Status: Normal Configured Capacity: 42140401664 (39.25 GB) 24576 (24 KB) Non DFS Used: 3717365760 (3.46 GB) DFS Remaining: 36258803712 (33.77 GB) DFS Used%: 0.00% DFS Remaining%: 86.04% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last Contact: Wed Jan 20 23:56:42 CST 2021 Last Block Report: Wed Jan 20 23:53:06 CST 2021 Num of Blocks: 0Copy the code

Log in to huawei Cloud. Open security group ports 9870 and 8088

accessxxx.xxx.xxx.xxx:9870/Replace with a public IP address

accessxxx.xxx.xxx.xxx:8088/Replacing public IP addresses

Reference:

Juejin. Cn/post / 684490…

Hadoop.apache.org/docs/stable…

The next article

Zookeeper+Hbase installation: juejin.cn/post/692065…

Panda Notebook Email:[email protected]