Zookeeper principle & Integrated Development environment construction

Preface – Theory

To be added

The deployment environment

The JDK 1.8.0 comes with _151

Zookeeper – 3.5.9

Kafka_2. 11 – against 2.4.1 scala version 2.11

10.0.20.181, 10.0.20.182, 10.0.20.183

Preparations for Deployment

Ping/No-secret login for the three servers. See Hadoop Installation preparations juejin.cn/post/694835…

! Note: all three machines synchronize their clocks. ! Note versioning issues.

Download the installation package kafka.apache.org/ and use the old XFTP method to transfer the installation package

Procedure Install ZooKeeper

unzip

Tar -xzvf kafka_2.11-2.1.1. TGZ -c /usr/local

Change the name

The mv kafka_2. 11 - against 2.4.1 kafka - against 2.4.1

The environment variable

Enter the vim/etc/profile

export KAFKA_HOME=/usr/local/kafka_211.-2.41.
export PATH=$PATH:$KAFKA_HOME/bin
Copy the code

Run source /etc/profile immediately

The configuration file

Into the CD/usr/local/kafka – against 2.4.1 / config

vim server.properties

Change a few variables

broker = 0/ broker = 1/ broker = 2Note Master/slave1/ slave2 listeners=PLAINTEXT:/ / 10.0.20.181:9092
advertised.listeners=PLAINTEXT:/ / 10.0.20.181:9092Information storage address log.dirs=/home/The number of projet/kafka/logs partitions can be changed depending on the item num. Partitions =3Zookeeper connection zookeeper. Connect =10.020.181.:2181.10.020.182.:2181.10.020.183.:2181Combined with the host. Nmae =10.020.181.
Copy the code

Listeners are the addresses of kafka’s true bind

12. Advertised are external listeners, and may be used if not advertised

Create information storage address CD /home/projet/kafka/, mkdir logs

Configure clusters Slave1 and Slave2

Copy master ZooKeeper across VMS to Slave1 and Slave2

SCP - r/usr/local/kafka - against 2.4.1 slave1: / usr/local

SCP - r/usr/local/kafka - against 2.4.1 slave2: / usr/local

Mkdir logs in CD /home/projet/kafka

Don’t forget to change the mapping between broker ID and host.name

Start/stop/View status/JPS/Create topic/ View/etc

Start the zookeeper,

Run the /usr/local/zookeeper-3.5.9 command to switch to zooKeeper

Start the bin/zkServer. Sh start

Start the kafka

Run the /usr/local/kafka-2.4.1 command to go to kafka

Start kafka bin/kafka-server-start.sh -daemon config/server.properties

-daemon Daemon process

Create a Topic single line pattern

Bin /kafka-topics. Sh --create --zookeeper 10.0.20.181:2181 --replication-factor 1 -- Partitions 3 --topic test_1

Create a Topic cluster pattern

Bin/kafka - switchable viewer. Sh -- -- create -- zookeeper 10.0.20.181:2181,10.0. 20.182:2181,10.0. 20.183:2181 - replication - factor 1 --partitions 3 --topic test_1

Replication-factor — number of partitions — 3 –topic

View topic details/view topic list

View topic details

Sh --zookeeper 10.0.20.181:2181 --topic test_1 --describe

View the list of topics

Sh --zookeeper 10.0.20.181:2181 --topic test_1 --list

Create consumers

Bin /kafka-console-consumer.sh --bootstrap-server 10.0.20.181:9092 --topic test_1 --from-beginning