Kafka Configuration 1– Install and configure Kafka for Windows

​

Kafka Configuration 1– Install and configure Kafka for Windows

Kafka Configuration 2– Configure SASl-plain authentication for Kafka under Windows

Kafka configuration 3– Configure the Kafka cluster under Windows

Kafka Configuration 4– Configure the SSL certificate for Kafka on Windows

Kafka configuration 5– Configure Kafka cluster +SASL+SSL on Windows

Kafka Configuration 6– Set and add SASL users or user permissions in Windows

1, download address: kafka.apache.org/downloads.h…

Tsinghua mirror: mirrors.tuna.tsinghua.edu.cn/apache/kafk…

Such as:

Mirrors.tuna.tsinghua.edu.cn/apache/kafk…

Mirrors.tuna.tsinghua.edu.cn/apache/kafk…

​

Since Kafka relies on the Java JDK and The ZooKeeper environment, you should first install both in order.

2.1. Install Java JDK (omitted) For example, add the name of the JAVA_HOME variable to the system variable and the value of the JAVA_HOME variable is D:\Net_Program\Net_Java PATH. Add %JAVA_HOME%\bin and %JAVA_HOME%\jre\bin

2.2, installing zookeeper 2.2.1, download address: website address: www.apache.org/dyn/closer…. Tsinghua mirror: mirrors.tuna.tsinghua.edu.cn/apache/zook…

Such as:Mirrors.tuna.tsinghua.edu.cn/apache/zook…

​

2.2.2 Installation Decompress the downloaded Zookeeper package to a directory, for example, D:\Net_Program\Net_Zookeeper directory: bin: executable script for kafka operation, including the Windows script config: Directory where the configuration file resides libs: dependency library directory site-docs:

2.2.3 Modify the configuration file. Copy zoo_sample. CFG from the decompressed conf directory and rename it zoo. CFG. Because zoo. CFG in conf path is loaded by default when the project is started.) Modify the dataDir configuration and add the dataLogDir configuration, dataDir is used to store data and dataLogDir is used to store logs, as shown below:

DataLogDir =D:/Net_Program/Net_Zookeeper/data-file dataLogDir=D:/Net_Program/Net_Zookeeper/ data-fileCopy the code

​

Or use Zookeeper built into Kafka

DataDir =D:/Net_Program/Net_Kafka/data-zookeeper/data-file Normally, log files are mounted on a separate disk partition dataLogDir=D:/Net_Program/Net_Kafka/data-zookeeper/data-logCopy the code

2.2.4 Starting ZooKeeper Run CMD as the administrator to go to the bin directory in the installation directory and run zkServer or double-click zkserver.cmd

We can also set it to a system variable by adding the system variable name ZOOKEEPER_HOME with the value D:\Net_Program\Net_Zookeeper

Add %ZOOKEEPER_HOME%\bin to Path, then run CMD as administrator, enter zkServer directly to start the service

Run the nc-vz 192.168.3.200 2181 command to check whether the Zookeeper service is enabled (netcat must be installed on the PC).

2.2.5. Set Zookeeper as a Windows service

Here we use the NSSM tool to set it up as a system service

Open the win64 folder of NSSM, such as D:\Net_ business software \Net_NSSM\ NSSM-2.24 \win64, then hold down Shift+ right click on the blank of the folder and select “Open Powershell window here”.

​

Then type.\ NSSM install to bring up the Settings form

In the popup form, in the Application TAB,

CMD, for example, D:\Net_Program\Net_Zookeeper\bin\zkServer.cmd

Startup Directory: After Path is selected, Startup directory is automatically populated

Argument: indicates the parameter used to start the service

Service name: indicates the Service name, for example, zookeeper-service

Zookeeper-service: zookeeper-service: zookeeper-service: zookeeper-service: zookeeper-service: zookeeper-service: zookeeper-service: zookeeper-service

​

​

Zookeeper has been built into Kafka. However, it is not recommended to use zooKeeper in Kafka

Zookeeper installation configuration reference: blog.csdn.net/skh2015java…

Zookeeper configuration parameters: blog.csdn.net/liereli/art…

​

Install Kafka. Decompress the downloaded Kafka package to a directory, for example, D:\Net_Program\Net_Kafka

Modify server.properties 3.1.1. Locate the server.properties file in the config directory, open it with notepad, and configure related information as required

Id =0 is used to listen for the port of the link. Num.net work.threads=3 # Number of threads used to process disk I/O Num.io. Threads =8 # Buffer size for sending sockets Socket, the send buffer. The bytes = 102400 # accept socket socket. The buffer size. The receive buffer. The bytes = 102400 # request socket buffer size Socket.request.max. bytes=104857600 # Kafka message path log.dirs=D:/Net_Program/Net_Kafka/kafka-data # topic Number of fragments in the current broker Num. Partitions = 1 # is used to restore and clean up the data under the data on the number of threads num. Recovery. Threads. Per. Data. The maximum dir = 1 # default message persistence time, 168 hours, 7 days log.retention. Hours =168 # Size of each segment in log file, Log. The default is 1 g segment. Bytes = 1073741824 # periodically check the file size of the time the retention. Check. Interval. Ms = 300000 # log clear whether open, general need not enabled, Log.cleaner. Enable =false # ZooKeeper cluster address Comma separated between multiple hostname1: port1, hostname2: port2, hostname3: port3 zookeeper. Connect = localhost: 2181 # zookeeper link timeout Zookeeper. Connection. A timeout. Ms = 6000 # partion buffer, news article number reach the threshold, Will trigger the flush to disk # log. Flush. Interval. The messages = 10000 # message buffer time, reach the threshold, Will trigger the flush to disk # log. Flush. Interval. Ms = 1000 # # # # # # # # # # # # # # # # # # # # # # # # # # # # # other # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # message to save a maximum of 20 m Message.max. byte=20971520 # Kafka saves the number of copies of the message, if a copy is invalid, Another can also continue to provide service. The default replication. The factor = 2 # the maximum number of directly take message up. The fetch. Max. Bytes = 5242880Copy the code

3.1.2, detailed configuration instructions reference: apache server of kafka series. The properties parameters configuration file: blog.csdn.net/lizhitao/ar… Kafka installation tutorial (detailed process) : blog.csdn.net/Poppy_Evan/…

3.2. Start the Kafka service. Run CMD as the administrator to locate the installation directory and run the following command

.\bin\windows\kafka-server-start.bat .\config\server.properties
Copy the code

​

Add %KAFKA_HOME%\bin and %KAFKA_HOME%\bin\ Windows to D:\Net_Program\Net_Kafka Path. Then run CMD as administrator and enter the following command to start the service

kafka-server-start D:\Net_Program\Net_Kafka\config\server.properties
Copy the code

Use the nC-vz 192.168.3.200 9092 command to verify that Kafka is enabled (netcat is required).

Set Kafka to serve Windows

Here we use the NSSM tool to set it up as a system service

Open the win64 folder of NSSM, such as D:\Net_ business software \Net_NSSM\ NSSM-2.24 \win64, then hold down Shift+ right click on the blank of the folder and select “Open Powershell window here”.

​

Then type.\ NSSM install to bring up the Settings form

In the popup form, in the Application TAB,

Bat directory, for example, D:\Net_Program\Net_Kafka\bin\ Windows \kafka-server-start.bat

Startup Directory: After Path is selected, Startup directory is automatically populated

Argument: indicates the parameter to start the service. Enter D:\Net_Program\Net_Kafka\config\server.properties

Service name: indicates the Service name, for example, kafka-service

Kafka-service: kafka-service: kafka-service: kafka-service: kafka-service

​

​

Run CMD as an administrator, navigate to the installation directory, and run the following command to create a theme named TestTheme1

.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TestTopic1
Copy the code

or

kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TestTopic1
Copy the code

3.3.2 Viewing a theme Run CMD as an administrator to locate the installation directory and run the following command to view the created theme

.\bin\windows\kafka-topics.bat --zookeeper localhost:2181 --list
Copy the code

or

kafka-topics --zookeeper localhost:2181 --list
Copy the code

​

Run CMD as the administrator (open a CMD form, that is, simulate the producer form), locate the installation directory, and run the following command

.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic TestTopic1
Copy the code

or

kafka-console-producer --broker-list localhost:9092 --topic TestTopic1
Copy the code

​

Run CMD as the administrator (open a new CMD form, that is, simulate the consumer form) to locate the installation directory. Bat –zookeeper localhost:2181 –topic TestTopic1 → Zookeeper is not a recognized option:

.\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic TestTopic1 --from-beginning
Copy the code

or

kafka-console-consumer --bootstrap-server localhost:9092 --topic TestTopic1 --from-beginning
Copy the code

​

3.3.5. Test sending and receiving

Switch to the producer form you just created, type enter, and then you can see the content of the message sent by the producer in the consumer form

​

4, Kafka visualization tools Download address: www.kafkatool.com/download.ht…

Such as:www.kafkatool.com/download2/k…

​

Download and install

Settings:

After installation, open the software, Tools→Settings→Topics, set Key and Message in Default Content Types on the right to String, and click OK button

​

You can also mouse click on a specific topic, select String for Key and Message in Content Types from the Properties panel on the right, and then click Update and Refresh

​

Refer to the article: kafka visualization client Tool (kafka Tool) the basic use of: www.cnblogs.com/frankdeng/p… Kafka Visualization Tool: www.cnblogs.com/miracle-lun…

5, the related documents to install and configure Kafka: blog.csdn.net/twypx/artic…

Kafka configure SSL authentication: blog.csdn.net/justry_deng… Generation of SSL certificates: blog.csdn.net/justry_deng…

Kafka distributed message system (SSL client) : www.tracefact.net/tech/109.ht… Certificate key conversion P12, JKS, CER, RFX PEM:www.jianshu.com/p/dff237d75…

​