Finishing the adjava.net lify. # / app /. / docs/hi…

Making: github.com/KinghooWei/… Includes PDF files and XMind files

Reprinted with note from: juejin.cn/post/693873…

The message queue

Usage scenarios

The decoupling

  • Publish subscribe message model

asynchronous

  • Processing time-consuming operation

Peak clipping

  • High concurrency

species

ActiveMQ

RabbitMQ

  • Low latency

RocketMQ

Kafka

disadvantages

The system availability decreases

System complexity enhancement

Consistency problem

High availability

RabbitMQ

  • Stand-alone mode
  • Common Cluster mode
  • Mirroring cluster mode

Kafka

  • Distributed architecture: Create topics that are divided into multiple partitions, each with a portion of the data stored on different brokers

  • Replica mechanism: All replicas of partitions are distributed on different machines. The leader reads and writes data

    • Write: The consumer writes to the leader, the leader writes to the local disk, and the follower actively pulls data from the leader. After all the followers are synchronized, the leader sends an ACK to the leader, and the leader returns a success message to the consumer
    • Read: Only after the leader receives an ACK can it be read by the consumer

Design of MQ

See Kafka, distributed architecture, scalability, increased throughput

Landing disk, sequential writes, without the addressing overhead of random disk reads and writes

High availability, multiple copies

Data 0 is lost

idempotence

Scenario: A restart occurs before the consumer can submit the offset

plan

  • Database: set the primary key to prevent dirty data; Check before insert, if yes, update
  • Redis: Natural idempotent
  • A globally unique ID is added to the data sent by the producer. Each consumption queries Redis according to the ID. If there is one, it will be discarded; if there is none, it will be written to Redis and processed

Reliability transmission

Kafka

  • Consumer: The consumer submits offset, but dies before processing it

    • You can handle offsets first and then commit them, but beware of idempotent issues

  • Kafka: The leader dies before the followers have synchronized their data

    • A partition has at least two followers
    • At least one follower keeps in touch with the leader
    • Data is written to all followers before the producer considers the write successful
    • If a write fails, infinite retry

RabbitMQ

  • producers

    • Transaction mechanism
    • Confirm mechanism

  • RabiitMQ

    • Persistence is combined with the producer’s confirm mechanism

  • consumers

    • Disable RabbitMQ automatic ack

sequential

Kafca

  • Consumers use multithreading to process data in order

    • All data with the same key goes to the same memory queue. For N threads, each thread consumes one memory queue

RabbitMQ

  • Multiple consumers consume data, breaking sequential processing

    • Split queues, one consumer per queue

Message backlog

Fix consumer, suspend existing consumer

New topic, partition is 10 times of the original

Write a consumer program that distributes data

Temporarily requisition 10 times as many machines to deploy consumers

RocketMQ

  • Improve consumption parallelism
  • Bulk consumption
  • Skip non-essential purchases
  • Optimize the per-message consumption process

Message expiration

Batch weighing guide