Ambari is deployed offline

Ambari introduction

Ambari is a top-tier open source project from the Apache Software Foundation. It is a tool for centrally deploying, managing, and monitoring Hadoop distributed clusters. But Hadoop is a broad concept, not just Hadoop (HDFS, MapReduce), but the Hadoop ecosystem (including Spark, Hive, HBase, SQoop, ZooKeeper, Flume, etc.). Ambari can make Hadoop big data software easier to use and easily integrate our own services so that Ambari can manage them uniformly.

Deployment: Automated deployment of Hadoop software, which can automatically handle dependencies between services and components (for example, HBase relies on HDFS, and NameNode needs to be started first when the DataNode is started).
Management: Start, stop, restart of Hadoop service components, multi-version management of configuration files.
Monitoring: The current state of the Hadoop service (component node survival, YARN task execution, etc.), the current state of the host (memory, hard disk, CPU, network, etc.), and the ability to customize alarm events.

The Ambari version corresponds to the table

HDP2.5 version is used this time

Cluster planning

Cluster hostname	Software installation
nn1	ambari-agent,namenode,datanode,zookeeper
nn2	ambari-agent,namenode,datanode,zookeeper
rm	ambari-server,resourcemanager,historyserver,datanode

Install Ambari – Server

Download the offline package you need first. These packages are uploaded to the node rm. Offline package required:

Hdp-utils-1.1.0.21-centos7.tar. gz hdp-util-1.1.0.21-centos7.tar. gz
Ambari -2.4.1.0-centos7.tar.gz (Ambari offline package)
Hdp-2.5.0.0 -centos7-rpm.tar.gz (official download link)
Centos-7-x86_64-DVD-1503-01. ISO (Centos7 Offline Package)
jdk

Centos7 64 system software requirements

yum and rpm
scp,curl,unzip,tar,and wget
openSSL
Python 2.7.x
JDK 1.8.0_77 (officially recommended, but I can use 1.8.0_11)
It is recommended to modify the maximum open file description to 10000, temporary changeulimit -n 10000

Permanent change, need to modify the configuration file, and need to restart, recommended and SELinux closed together, then do restart action.

Vi/etc/security/limits. Conf add * soft nofiles 10000 * hard nofiles 10000

The language must be English by default

Install Ansible (optional)

Note: For the following steps, use the Ansible command. If you do not have Ansible installed, use the original command. Ansible all-m command-a ‘systemctl stop firewalld’ is equivalent to executing the command systemctl stop firewalld on every host

Ansible installation and operation

Configure SSH free secrecy

Modify Hosts to increase your cluster information, same for each server

192.168.0.135 nn1. Ambari Nn1 192.168.0.136 nn2. Ambari Nn2 192.168.0.137 rm. Ambari Rm

On the RM node, execute the command ssh-keygen, directly press Enter four times, and then execute again

ssh-copy-id nn1
ssh-copy-id nn2
ssh-copy-id rm

Turn off the firewall

ansible all -m command -a 'systemctl stop firewalld'

ansible all -m command -a 'systemctl disable firewalld'

Close the Selinux

Change /etc/selinux/config to selinux =disable, reboot is required

Install the JDK

1. Unpack

cd /opt
tar -zxvf  jdk-8u111-linux-x64.tar.gz

2. Add configuration to /etc/profile

Export JAVA_HOME=/opt/jdk1.8.0_111 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

3. Copy JDK packages and configuration files to NN1 and NN2 nodes

Scp-r jdk1.8.0_111 nn2:/opt/ SCP /etc/profile nn2: /etc/scp-r jdk1.8.0_111 nn1:/opt/ SCP /etc/profile nn1:/opt/ SCP /etc/profile nn1: /etc/profile

4. Verify the java-version

Mount the Centos7 source offline

Create a new directory to mount (for later convenience of HTTPD source, Mount -t iso9660-o loop /opt/ centos-7-x86_64-dvd-1511.iso: mount -t iso9660-o loop /opt/ centos-7-x86_64-dvd-1511.iso: mount -t iso9660-o loop /opt/ centos-7-x86_64-dvd-1511.iso /var/ WWW/HTML /centos7 3. Back up other Repo files and build the offline source

cd /etc/yum.repos.d
mkdir backup
mv Centos-* backup
touch centos-media.repo

Write the following in centos-media.repo

[centos7 - media] name = Centos Linux 7.0 baseurl = file:///var/www/html/centos7 enabled = 1 gpgcheck = 0

4. Verify that the offline source is available.

yum clean all
yum makecache

Configure Ambari, HDP offline source

1. Configure the path

cd /var/www/html
mkdir ambari
mkdir hdp

2. Unzip the package to the corresponding path

Tar-zxvf /opt/Ambari/ Ambari 2.4.1.0-centos7.tar.gz-c /var/ WWW/HTML /Ambari/ tar-zxvf /opt/Ambari/ Ambari 2.4.1.0-centos7.tar.gz-c /var/ WWW/HTML /Ambari/ tar-zxvf /opt/Ambari/ hdp-util-1.1.0.21-centos7.tar.gz /var/ WWW/HTML /HDP/tar-zxvf /opt/Ambari/ hdp-util-1.1.0.21-centos7.tar.gz /var/ WWW/HTML /HDP/tar-zxvf /opt/Ambari/ hdp-util-1.1.0.21-centos7.tar.gz  -C /var/www/html/hdp/

3. Build an offline source

3.1. Build on RM node

Ambari offline source vi/etc/yum repos. D/ambari. Repo

[ambari-2.4.1.0] name= ambari-2.4.1.0-Updates BaseURL =http://rm/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22 GPGCheck =0 #gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1

HDP offline source vi/etc/yum repos. D/HDP. Repo

[HDP 2.5] name = HDP 2.5 baseurl gpgcheck = 0 = http://rm/hdp/HDP/centos7 enabled = 1

Hdp-utils offline source vi /etc/yum.repos.d/hdp-utils. Repo

[hdp-utils-1.1.0.21] name= hdp-utils-1.1.0.21 BaseURL =http://rm/hdp/HDP-UTILS-1.1.0.21/repos/centos7 Enabled =1 GPGCheck =0

Install the Apache Httpd

1, install yum install HTTPD – y 2, modify configuration vi/etc/HTTPD/conf/HTTPD. Conf

Change the red color in the figure to ServerName RM :80

<Directory />
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

Open a browser on your computer, type rm/centos7 (if the host is configured properly), and see the following image: That means success.

Install the time synchronization tool chrony

ansible all -m yum -a "name=chrony"

Modify the chrony configuration file on the RM node

To modify the configuration files on the NN1 and NN2 nodes, simply add more servers to synchronize

Ansible all-m command-a ‘systemctl start chronyd’

Check yourself if the time has been synchronized.

Install ambari – server

1. Install yum install ambari-server-y. 2. Install ambari-server setup

Most people just follow the default.

[root@rm ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):root Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter  choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /opt/jdk1.8.0_111 Validating JDK on Ambari Server... done. Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (1): 1 Database name (ambari): ambari Postgres schema (ambari): ambari Username (ambari): ambari Enter Database Password (bigdata): Invalid characters in password. Use only alphanumeric or _ or - characters Enter Database Password (bigdata): Default properties detected. Using built-in database. Configuring ambari database... Checking PostgreSQL... Running initdb: This may take up to a minute. Initializing database ... OK About to start PostgreSQL Configuring local database... Connecting to local database... done. Configuring PostgreSQL... Restarting PostgreSQL Extracting system views... Ambari - admin - 2.4.1.0.22. Jar... Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully.

3. Start Ambari-Server Start

Install ambari – agent

As a condition for ambari-agent:

Firewall off
Selinux closed
The JDK installation
Chrony installation
Since this is an offline environment, you also need to configure the offline source

Because of the previous four conditions, ambari-server was already installed when it was installed.

All that is left to do is configure the offline source

Remove the default repo rm -f from /etc/yum.repos.d/*.repo

Copy the RM’s offline source to another node

scp /etc/yum.repos.d/ambari.repo nn1:/etc/yum.repos.d/
scp /etc/yum.repos.d/ambari.repo nn2:/etc/yum.repos.d/
scp /etc/yum.repos.d/hdp.repo nn2:/etc/yum.repos.d/
scp /etc/yum.repos.d/hdp.repo nn1:/etc/yum.repos.d/
scp /etc/yum.repos.d/hdp-utils.repo nn1:/etc/yum.repos.d/
scp /etc/yum.repos.d/hdp-utils.repo nn2:/etc/yum.repos.d/

3.3 Verify on NN1 and NN2

yum makecache

A new Ambari-Agent

You need to reconfigure the pre-environment mentioned earlier.

You also need to modify the ambari-server’s /etc/hosts to include the new machine and execute ssh-copy-id $new_hostname

Install the cluster on ambari-web

Open a browser and enter the IP/HostName:8080 of the node where ambari-server is located. The next step is to operate directly in the interface.

Just look out for a few potholes.

I forgot to take a screenshot here, so here’s a link to deploy a Hadoop 2.x cluster for reference.

When installing, it will report an error with the wrong Snappy package

resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install snappy-devel' returned 1. Error: Package: Snappy -devel-1.0.5-1.el6.x86_64 (hdp-utils-1.1.0.21) Requires: snappy(x86-64) = 1.0.5-1.el6 Installed: Requires: snappy(x86-64) = 1.0.5-1.el6 Installed: Snappy -1.1.0-3.el7.x86_64 (@anaconda) snappy(x86-64) = 1.1.0-3.el7 Available: Snappy -1.0.5-1.el6.x86_64 (hdp-utils -1.1.0.21) snappy(x86-64) = 1.0.5-1. El6 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest

Removal of Snappy Ansible All-M Yum-A ‘Name = Snappy State = Removed Snappy Ansible All-M Yum-A ‘Name = Snappy – Devel ‘

Online existing resources Docker install Ambari

This is very convenient, basically 10 minutes can set up a test cluster, but the fly in the oint is that the author’s source is not offline, to set up a cluster requires a network environment.

GitHub Docker-Ambari can be downloaded from the GitHub Docker-Ambari website. According to the README document on the website, you can simply deploy it

Ambari-functions or source ambari-functions' 3.3 Amb-Start-Cluster 3! [] (.. /images/ambari/ambari_docker_github_result.jpg)

Problems remain

The author mentioned in the README that Ambari containers started by Ambari -function are using bridge networking. This means that you will not be able to communicate with containers directly from host unless you specify the route to containers. You can do this with:

# Getting the IP of docker-machine or boot2docker docker-machine ip <name-of-docker-vm> # or boot2docker ip # Setting up The sudo route add-net 172.17.0.0/16 <docker-machine or boot2docker-> # lg_block Sudo route add-net 172.17.0.0/16 192.168.99.100

172.17.0.0 can be connected to the network by configurable route, that is to say, the author did not map the port, I configured my own network route, but can not connect.

After that, you can start ambari-server by modifying the command that Docker starts ambari-server. Then, you can access the ambari-server image created by the author by entering host IP: map port number. As shown in figure:

Docker builds Ambari offline

1. Put the offline sources of Ambari and HDP on the host and extract them. 2

The preparation documents are as follows:

ambari.repo           
Dockerfile                 
hdp.repo                   
hdp-utils.repo             
httpd.conf                 
jdk-8u111-linux-x64.tar.gz 
supervisord.conf

Dockerfile content

# ADD JDK /usr/local/ SRC/ENV FROM centos MAINTAINER linjk # ADD JDK / 8u111-linux-x64.tar.gz /usr/local/ SRC/ENV JAVA_HOME = / usr/local/SRC/jdk1.8.0 _111 ENV PATH = $PATH: $JAVA_HOME/bin ENV $JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar :$JAVA_HOME/lib/tools.jar Fast \ && yum install HTTPD openssh-server net-tools chrony-y # install supervisor RUN yum -y install python-setuptools \ &&  easy_install supervisor \ && mkdir -p /etc/supervisor COPY supervisord.conf /etc/supervisor/ COPY httpd.conf /etc/httpd/conf/httpd.conf RUN mkdir -p /var/www/html/centos/ \ && mkdir -p /var/www/html/ambari/ \ && mkdir -p /var/ WWW/HTML/HDP/COPY *.repo /etc/yum.repos. / # install -y Mariadb -Mariadb -server: /var/ WWW/HTML/HDP/COPY *.repo /etc/yum.repos. / # install -y Mariadb -server: /var/ WWW/HTML/HDP/COPY * mysql-connector-java EXPOSE 22 80 9001 8080 3306 CMD supervisord -c /etc/supervisor/supervisord.conf

Ambari. ‘content

[ambari-2.4.1.0]
name=ambari-2.4.1.0 - Updates
baseurl=http://ambari-server/ambari/AMBARI-2.4.1.0/centos7/2.4.1.0-22
gpgcheck=0
#gpgkey=http://public-repo-1.hortonworks.com/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

HDP. ‘content

[hdp-2.5] name= hdp-2.5 BaseUrl =http://ambari-server/hdp/HDP/centos7 Enabled =1 GPGCheck =0

HDP – utils. ‘content

[HDP UTILS - 1.1.0.21] name = HDP UTILS - 1.1.0.21 baseurl = http://ambari-server/hdp/HDP-UTILS-1.1.0.21/repos/centos7 enabled = 1  gpgcheck=0

Httpd.conf modified contents

ServerName ambari-server:80

#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other
# <Directory> blocks below.
#
<Directory />
  Options FollowSymLinks
  AllowOverride None
  Order allow,deny
  Allow from all
</Directory>

The supervisor content

unix_http_server]
file=/var/run/supervisor.sock

[supervisord]
nodaemon=true

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[inet_http_server]
port=*:9001

[program:sshd]
command=/usr/sbin/sshd -D

[program:httpd]
command=/usr/sbin/httpd

3, build image ambari – server (before compression bag on/opt/ambari and/opt/HDP /) docker run – d – P – v/opt/ambari / : / var/WWW/HTML/ambari / – v /opt/hdp/:/var/www/html/hdp/ -h ambari-server –name as ambari-server /usr/sbin/init

Enter the container to start and configure mariaDB

Mysql_secure_installation mysql_mysql_installation mysql_secure_installation mysql_mysql_installation mysql_mysql_installation mysql_mysql_installation mysql_mysql_installation mysql_mysql_installation mysql_mysql_installation

An interactive interface will appear, basically press enter all the way to set the password screen

Enter current password for root (Enter for none):< -- When running for the first time, press Enter to Set password for root? [Y/n] [Y/n] < -- If you want to set password for root, type "Y" and "enter" or "enter"; < -- Enter the password you set again to Remove anonymous users? [Y/n] < -- Disallow Root Login Remotely? [Y/n] < -- Disallow root remote login, press enter, Remove test database and access to it? [Y/n] < -- Do you want to delete test database, enter Reload privilege tables now? [Y/n] < -- Whether to reload the permission table, press Enter

Mysql > configure character set vi /etc/my.cnf for mariaDB

create database ambari;
use ambari;
CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
CREATE USER 'ambari'@'ambari-server' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'ambari-server';

5. Install ambari-server, configure and start 5.1, install yum install ambari-server 5.2, configure ambari-server setup

Interaction interface appears

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)? y
Enter user account for ambari-server daemon (root):root
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid o
Path to JAVA_HOME: /usr/local/src/jdk1.8.0_111
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (3): 3
Hostname (ambari-server):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.4.1.0.22.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

Use ambari; use ambari;

use ambari;
source  /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

Verify that any Ambari configuration tables have been generated

show tables;

7. Verify that the $host IP: map port is open in the browser

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Ambari environment setup (offline setup)

Ambari is deployed offline

Ambari introduction

The Ambari version corresponds to the table

Cluster planning

Install Ambari – Server

Centos7 64 system software requirements

Install Ansible (optional)

Configure SSH free secrecy

Turn off the firewall

Close the Selinux

Install the JDK

Mount the Centos7 source offline

Configure Ambari, HDP offline source

Install the Apache Httpd

Install the time synchronization tool chrony

Install ambari – server

Install ambari – agent

A new Ambari-Agent

Install the cluster on ambari-web

When installing, it will report an error with the wrong Snappy package

Online existing resources Docker install Ambari

Problems remain

Docker builds Ambari offline

Ambari environment setup (offline setup)

Ambari is deployed offline

Ambari introduction

The Ambari version corresponds to the table

Cluster planning

Install Ambari – Server

Centos7 64 system software requirements

Install Ansible (optional)

Configure SSH free secrecy

Turn off the firewall

Close the Selinux

Install the JDK

Mount the Centos7 source offline

Configure Ambari, HDP offline source

Install the Apache Httpd

Install the time synchronization tool chrony

Install ambari – server

Install ambari – agent

A new Ambari-Agent

Install the cluster on ambari-web

When installing, it will report an error with the wrong Snappy package

Online existing resources Docker install Ambari

Problems remain

Docker builds Ambari offline

Related Posts

SQOOP learning summary

A must-see for ETL engineers! Super practical task optimization and breakpoint execution scheme

Hive Modeling Analysis