What is Kafka
Kafka is a distributed publish/subscribe message queue and a distributed event flow platform. It can be used not only as message queue, but also as data pipeline and flow analysis. The biggest use of Kafka today is message queues.
The mainstream message queues in the market include RabbitMQ, ActiveMQ, Kafka, etc. RabbitMQ and ActiveMQ are mainly queues in Java applications, while Kafka is mainly used in big data scenarios.
The main application scenarios of message queues are as follows: peak clipping, traffic limiting, decoupling, and asynchronous communication.
(2) the realization mode of message queue
There are two main modes of message queue implementation, one is called point-to-point mode: after the producer sends the message to the queue, the consumer retrives the message from the queue and consumes it. This pattern ensures that a message is consumed by a single consumer only once, and that no message can be repeated.
The second is the publish/subscribe model, which Kafka uses. In the publish/subscribe pattern, there are multiple topic topics in the queue, producers send messages to the topics in the queue, and consumers can subscribe to a topic to consume data. And after consumers consume the data, the data will not be deleted.
(3) Kafka architecture design
As mentioned earlier, Kafka is primarily used in big data environments, which inevitably involve huge amounts of data. When it comes to terabytes of data, the current optimal design architecture is divide and conquer, that is, spread the data across different machines, and Kafka applies this design approach.
A Kafka cluster divides a Topic into partitions (partitions), and each Topic can specify the number of replicas that can be assigned to different machines. At the same time, in order to improve the consumption ability of consumers, the way of consumption group is used to make consumers in the consumption group consume in parallel. In order to improve reliability, copy is adopted to ensure availability. Use Zookeeper to record configuration information.
Fast installation of Kafka
By default, Kafka configuration information is stored in ZooKeeper. Therefore, zooKeeper must be installed and started in advance. Otherwise, Kafka reports a connection failure.
Kafka 3.0: kafka_2.13-3.0.0. TGZ: Kafka 3.0: kafka_2.13-3.0.0. TGZ
Downloads.apache.org/kafka/3.0.0…
Once the download is complete, upload the file to your Linux server and the installation can begin
Tar -xzf kafka_2.13-3.0.0. TGZ mv kafka_2.13-3.0.0 kafkaCopy the codeKafka /config/server.properties: / TMP/log.dirs: / TMP/server.properties Change the zooKeeper address to a customized directory, and change the ZooKeeper address to your own.
log.dirs=/usr/local/kafka/datas
zookeeper.connect=localhost:2181
Copy the codeStart kafka under kafka:
./bin/kafka-server-start.sh -daemon ./config/server.properties
Copy the codeIn the kafka/logs directory, look at server.log. If started is displayed, it indicates success. Otherwise, modify the configuration or command based on the error.
Kafka command line operation
Kafka can be divided into producers, consumers and services. This section mainly introduces some command line operations of Kafka itself kafka-topics. The common command line parameters are given in the form of tables
| parameter | describe |
|---|---|
| –boostrap-server | Connect the Kafka |
| –topic | The topic name of the operation |
| –create | Create a topic |
| –delete | Delete the topic |
| –alter | Change the topic |
| –list | View all topics |
| –describe | View details about a topic |
| –partitions | Set the number of partitions |
| –replication-factor | Example Set the number of copies of a partition |
5.1 create a Topic
Use the following command to create a Topic named testTopic with 1 partitions and 1 copies
Sh --bootstrap-server 127.0.0.1:9092 --topic testTopic --create --partitions 1 --replication-factor 1Copy the code5.2 check the Topic
Describe allows you to view details about a Topic:
/bin/kafka-topics. Sh --bootstrap-server 127.0.0.1:9092 --list./bin/kafka-topics --topic testTopic --describeCopy the code5.3 remove the Topic
The delete operation is the same as above:
Sh --bootstrap-server 127.0.0.1:9092 --topic testTopic --delete./bin/kafka-topicsCopy the code5.4 Simple message production and consumption
After creating a Topic, you can simply produce and consume messages. After creating a Topic, send a message from the producer’s command line:
/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --topic testTopicCopy the codeYou can then type some message, such as Hello World
Use the consumer’s command line consumption message in another session:
./bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic testTopic --from-beginning
Copy the code(6) Summary
This article mainly introduces what Kafka is, Kafka architecture, Kafka installation and basic use. There will be more kafka articles coming up, I’m Fish boy, and I’ll see you next time