Cloudera Manager

Cloudera Manager is divided into two parts: CDH and CM.

CDH is short for Cloudera Distribution Hadoop. As the name implies, it is the version of Hadoop published by Cloudera, which encapsulates Apache Hadoop and provides all services of Hadoop. These include HDFS,YARN,MapReduce, and various related components:HBase, Hive, ZooKeeper,Kafka, and more.

CM is the abbreviation of Cloudera Manager, which is the management platform of CDH, mainly including CM Server and CM Agent. Through CM, CDH can be configured, monitored, alerted, log view, dynamically add and delete various services and so on.

First, preparation work

The environment

JDK:1.8 CentOS :7.3 Operating System: CentOS 6 JDK: 1.7.0_80 Operating System: 1.7.0_80 Cloudera manager - centos7 - cm5.12.1 _x86_64. Tar. Gz CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel. Sha1 manifest. The json

Cloudera Manager download directory http://archive.cloudera.com/cm5/cm/5/

CDH download directory http://archive.cloudera.com/cdh5/parcels/5.12.1/

The manifest. Json download http://archive.cloudera.com/cdh5/parcels/5.12.1/manifest.json

Parcel packets associated with CHD5 are placed in the main node’s /opt/cloudera/parcel-repo/ directory

Cdh-5.12.1-1. Cdh5.12.1.p0.3 -el7. Parcel. Sha

In this paper, all packages, baidu network backup link: http://pan.baidu.com/s/1eSALjAq password: VQHG

Cdh-5.12.1-1.cdh5.12.1.p0.3-el6. Parcel file will be downloaded again

This article adopts offline installation method, online installation method please refer to the official document

The host name The IP address Installation services
node1 (Master) 192.168.252.121 JDK, Cloudera-Manager, MySQL
node2 (Agents) 192.168.252.122 The JDK, cloudera manager
node3 (Agents) 192.168.252.123 The JDK, cloudera manager
node4 (Agents) 192.168.252.124 The JDK, cloudera manager
node5 (Agents) 192.168.252.125 The JDK, cloudera manager
node6 (Agents) 192.168.252.126 The JDK, cloudera manager
node7 (Agents) 192.168.252.127 The JDK, cloudera manager

II. System environment construction

1. Network configuration (all nodes)

Change the hostname

The command format

hostnamectl set-hostname <hostname>

Modify all nodes in turn [1-7]

hostnamectl set-hostname node1

Restart the server

reboot

Modifying the mapping relationship

1. Modify the /etc/hosts file for all nodes and add the following

$ vi /etc/hosts

2. Check the contents of the modified /etc/hosts file

[root@node7 ~]# cat /etc/hosts 127.0.0.1 localhost localhost. Localhost 4 localhost4.localdomain4 ::1 Localhost localhost localhost6 localhost6.localdomain6 192.168.252.121 node1 192.168.252.122 node2 192.168.252.123 node3 192.168.252.124 node4 192.168.252.125 node5 192.168.252.126 node6 192.168.252.127 node7

2, SSH password free login

1. Comment out the following options in /etc/ssh/sshd_config file of Node1

vi /etc/ssh/sshd_config 
PubKeyAuthentication Yes # Enable public key authentication

2. Copy and send the modified /etc/ssh/sshd_config of the cluster Node1 to each node of the cluster by SCP command

for a in {2..7} ; do scp /etc/ssh/sshd_config node$a:/etc/ssh/sshd_config ; done

3. Generate public and private keys

1. Enter the command ssh-keygen-t rsa-p “in each node of the cluster to generate Key and press Enter

ssh-keygen -t rsa -P ''

4. Enter the command on the Node1 node of the cluster

Put the public key of each node in the cluster id_rsa.pub into its own authentication file authorized_keys;

for a in {1.. 7}; do ssh root@node$a cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys; done

5. Enter the command on the Node1 node of the cluster

/root/.ssh/authorized_keys’ : /root/.ssh/authorized_keys’ : /root/.ssh/authorized_keys

for a in {1.. 7}; do scp /root/.ssh/authorized_keys root@node$a:/root/.ssh/authorized_keys ; done

6. Enter commands for each node in the cluster

Connect and restart the SSH service

sudo systemctl restart sshd.service

7. Verify that SSH has no secret login

Open another window to test whether the secret login

For example: in node3

ssh root@node2

The exit exit

3. Turn off the firewall

systemctl stop firewalld.service

4. Close SELinux

To view

[root@node1 ~]# getenforce
Enforcing
[root@node1 ~]# /usr/sbin/sestatus -v
SELinux status:

temporarily closed

## Set SELinux to Permissive ## SetEnforce 1 Set SELinux to enforcing

Permanent ban

vi /etc/selinux/config

Change SELINUX=enforcing to SELINUX=disabled

The Settings need to be restarted to take effect

Node1: /etc/selinux-config: /etc/selinux-config: /etc/selinux-config: /etc/selinux-config: /etc/selinux-config: /etc/selinux-config

for a in {2.. 7}; do scp /etc/selinux/config root@node$a:/etc/selinux/config ; done

Restart all nodes

reboot

5. Install JDK

To download JDK 1.8 for Linux, please go to the JDK installation file

I in baidu cloud disk under the link: http://pan.baidu.com/s/1jIFZF9s password: u4n4

Upload to /opt directory

Unpack the

CD /opt tar ZXVF jdk-8u144-linux-x64.tar.gz mv jdk1.8.0_144/ /lib/ JVM

Configure environment variables

vi /etc/profile
#jdk
export JAVA_HOME=/lib/jvm
export JRE_HOME=${JAVA_HOME}/jre   
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib   
export PATH=${JAVA_HOME}/bin:$PATH

Enables environment variables to take effect

source /etc/profile

validation

[root@localhost ~]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

6. Set NTP

NTP is installed on all nodes

yum install ntp

Set the synchronous

Ntpdate -d 182.92.12.11

7. Install and configure MySQL

The primary node installs MySQL

MySQL relies on the libaio library

yum search libaio
yum install libaio

Download, unzip, rename

Usually unzip it in /usr/local/mysql

Mysql = ‘/usr/local/mysql-5.7.19-linux-glibc2.12-x86_64’; mysql = ‘/usr/local/mysql-5.7.19-linux-glibc2.12-x86_64’

CD/opt/wget tar ZXVF - https://dev.mysql.com/get/Downloads/MySQL-5.7/mysql-5.7.19-linux-glibc2.12-x86_64.tar.gz /opt/mysql-5.7.19-linux-glibc2.12-x86_64.tar.gz -c /usr/local/ mv /usr/local/mysql-5.7.19-linux-glibc2.12-x86_64/ /usr/local/mysql

1. Create new user groups and users

groupadd mysql
useradd mysql -g mysql

2. Create directories and authorize them

cd /usr/local/mysql/ 
mkdir data mysql-files
chmod 750 mysql-files
chown -R mysql .
chgrp -R mysql .

MySQL > initialize MySQL

Bin /mysqld --initialize --user=mysql # mysql 5.7.6 and up

Mysql > create password for mysql

[Note]root@localhost Generate temporary password:
; b; s;) /rn6A3, that is,
root@localhost:String after 

2017-09-24T08:34:08.643206Z 1 [Note] A temporary password is generated for root@localhost: D<qha)5gtr<!

5. Grant read and write permissions

chown -R root .
chown -R mysql data mysql-files

6. Add the MySQL startup script to the system server

cp support-files/mysql.server /etc/init.d/mysql.server

7. Grant read and write permissions to the log directory

mkdir /var/log/mariadb
touch /var/log/mariadb/mariadb.log
chown -R mysql:mysql /var/log/mariadb

8. Modify the/etc/my CNF

vi /etc/my.cnf

Modify the socket under the path/mysqld group, comment out the/var/lib/mysql/mysql. The sock, plus one behavior TMP/mysql. Soc

[mysqld]
datadir=/var/lib/mysql
#socket=/var/lib/mysql/mysql.sock
socket=/tmp/mysql.sock

9. Start MySQL server

service mysql.server start

or

/usr/local/mysql/support-files/mysql.server start

10. Log in to MySQL

/usr/local/mysql/bin/mysql -uroot -p
Enter password:

If you do not know the password, how to initialize the temporary password is mentioned in Step 4

MySQL > set password for MySQL

MySQL > set password for MySQL

mysql> ALTER USER   'root'@'localhost' identified by 'mima';
mysql> flush privileges;

12. Enable remote login

mysql> grant all privileges on *.*  to  'root'@'%'  identified by 'mima'  with grant option;
mysql> flush privileges;
mysql> exit;

8. Download the dependency packages

yum -y install chkconfig
yum -y install bind-utils
yum -y install psmisc
yum -y install libxslt
yum -y install zlib
yum -y install sqlite
yum -y install cyrus-sasl-plain
yum -y install cyrus-sasl-gssapi
yum -y install fuse
yum -y install portmap
yum -y install fuse-libs
yum -y install redhat-lsb

Cloudera Manager Server & Agent installation

1. Install CM Server & Agent

On all nodes, create /opt/cloudera-manager

mkdir /opt/cloudera-manager

Upload cloudera-manager-centos7-cm5.12.1_x86_64.tar.gz to node1 node /opt/

/opt/cloudera-manager create /opt/cloudera-manager create /opt/cloudera-manager create /opt/cloudera-manager create /opt/cloudera-manager create /opt/cloudera-manager

for a in {2.. 7}; do scp /opt/cloudera-manager-*.tar.gz root@node$a:/opt/ ; done

Unzip and install Cloudera Manager Server & Agent on all Server and Agent nodes

cd /opt
tar xvzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

Create user cloudera-scm (all nodes)

Cloudera-SCM User Description

Cloudera Manager Server and managed services are configured to use the user account cloudera-scm by default, creating a user with this name is the simplest approach. This created user, is used automatically after installation is complete.

The Cloudera Manager server and managed services are configured to use the user account Cloudera-SCM by default, and creating a user with this name is the easiest way to do so. Create a user and use it automatically after installation.

Execution: Create Cloudera-SCM users on all nodes

Useradd --system --home=/opt/cloudera-manager/cm-5.12.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3. Configure CM Agent

Modify the node1 node

/opt/cloudera-manager/cm-5.12.1/etc/cloudera-scm-agent/config.ini Server_host for master node

CD/opt/cloudera manager/cm - 5.12.1 the/etc/cloudera - SCM - agent/vi config. Ini

Modified node1 node during node1 operation (copy to all nodes)

for a in {1.. 7}; Do the SCP/opt/cloudera manager/cm - 5.12.1 the/etc/cloudera - SCM - agent/config. Ini Root @ node $a: / opt/cloudera manager/cm - 5.12.1 the/etc/cloudera - SCM - agent/config. Ini. done

4. Configure the CM Server database

Initialize CM5 database on primary node node1:

Download the MySQL driver package

CD/opt/cloudera manager/cm - 5.12.1 / share/wget CMF/lib http://maven.aliyun.com/nexus/service/local/repositories/hongkong-nexus/content/Mysql/mysql-connector-java/5.1.38/mysql- Connector - Java - 5.1.38. Jar

MySQL > start MySQL

service mysql.server start
CD /opt/cloudera-manager/cm-5.12.1/share/ CMF /schema/./scm_prepare_database.sh MySQL cm-h node1-uroot-pmima --scm-host node1 scm scm scm

See the following information, congratulations, the configuration is fine

[main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

Format:

Scm_prepare_database.sh mysql cm-h <hostName >-u <username >-p <password> --scm-host <hostName> SCM SCM SCM SCM SCM SCM SCM SCM SCM SCM SCM Database type database server user name password -- scm-host Cloudera_Manager_Server node...

5. Create the Parcel directory

/opt/cloudera/parcel-repo /opt/cloudera/parcel-repo

The file will be downloaded

CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel. Sha manifest. Json

Copy to this directory.

mkdir -p /opt/cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
cd /opt/cloudera/parcel-repo

Rename, cdh-5.12.1-1.cdh5.12.1.p0.3-el7. Parcel. Sha1 Otherwise, the system will re-download cdh-5.12.1-1.cdh5.12.1.p0.3-el7

Mv CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel. Sha1 CDH 5.12.1-1. Cdh5.12.1. P0.3 - el7. Parcel. Sha

/opt/cloudera/parcels

mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

6, start CM Manager&Agent service

Note that the MySQL service is started and the firewall is down

Execute on node1 (master) :

Server

/ opt/cloudera manager/cm - 5.12.1 the/etc/init. D/cloudera - SCM - server start

Execute in node2-7 (Agents) :

Agents

/ opt/cloudera manager/cm - 5.12.1 the/etc/init. D/cloudera - SCM - agent start

Visit http://Master:7180 If you can access (user name, password: admin), the installation is successful.

It takes some time for the Manager to launch successfully, and it takes some time for the corresponding tables to be created in the database.

IV. CDH5 installation

After CM Manager && Agent has successfully started, login to the front page for CDH installation and configuration.

The free version of CM5 has removed the 50 node limit.

After each Agent node starts normally, the corresponding node can be seen in the host list currently managed.

Select the node you want to install and click Continue.

If the Parcel is configured correctly, the download will be completed in a moment, and you will have to wait for the Parcel to be allocated. It may take about 10 minutes, depending on the internal network speed.

(If there is a problem with the local Parcel, recheck that steps 3 and 5 are configured correctly.)

If the Parcel is configured correctly, the download will be completed in a moment, and you will have to wait for the Parcel to be allocated. It may take about 10 minutes, depending on the internal network speed.

Have a problem

Next up is the server check. You may encounter the following problems:

Cloudera recommends setting /proc/sys/vm/swappiness to a maximum of 10. It is currently set to 30.

Use the sysctl command to change this setting at run time and edit /etc/sysctl.conf to save the setting after reboot.

You can proceed with the installation, but Cloudera Manager may report that your host is not performing well due to switching. The following hosts will be affected: node[2-7]

echo 0 > /proc/sys/vm/swappiness

Problem two has enabled transparent large page compression, which can cause significant performance problems. Please run the echo never > / sys/kernel/mm/transparent_hugepage/defrag and echo never > / sys/kernel/mm/transparent_hugepage/enabled To disable this setting, then add the same command to an initialization script such as /etc/rc.local to set when the system is restarted. The following hosts will be affected: node[2-7]

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Five, the script

MySql create database & delete database

1, MySQL build and delete libraries

amon

create database amon DEFAULT CHARACTER SET utf8; 
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';

hive

create database hive DEFAULT CHARACTER SET utf8; 
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

oozie

create database oozie DEFAULT CHARACTER SET utf8; 
grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

Contact

  • Author: Peng Lei
  • Reference: http://www.ymq.io
  • Email：[email protected]
  • Copyright belongs to the author, please indicate the source
  • WeChat: pay attention to public accounts, search cloud database, focus on the development of technology research and knowledge sharing