1. Zero copy
Kafka works primarily through two zero-copy technologies: Mmap and Sendfile
Zero copy does not mean that no copies are needed, but that the number of unnecessary copies is reduced, thus reducing the extra overhead.
Traditional data transmission:
As you can see from step 2 and step 3, the kernel and user space copies are not helpful but more expensive, so the zero copy is introduced. It is possible to copy directly from the kernel buffer to the network card buffer, skipping the CPU copy:
mmap
Mmap (Memory Mapped Files) deals with network data drop. It maps disk files to memory and then modifies the contents of the files by modifying memory.
Note that this mode requires flush to actually save data to disk. Kafka provides a parameter (producer.type) to control whether or not to flush actively. There are two values — sync and async. Sync calls flush immediately after a write is written and returns, and async returns without calling flush.
sendfile
It deals with the transfer of data from disk to network. The operating system reads disk data into the memory cache, sends it directly to the network adapter cache, and then sends network data.
2. Sequential read and write
When kafka writes data, it only appends data backward. It does not allow modifying existing data. Therefore, the order of data within each partition is guaranteed.
If there are too many topics or partitions, kafka may frequently switch partitions, and sequential reads and writes cannot be guaranteed. An Ali Middleware article, Kafka vs. RocketMQ — The Impact of multiple Topics on performance stability, tests this and concludes that when the number of topics exceeds 8, the time fluctuates dramatically.
3, PageCache
PageCache is a system-level cache that returns data as soon as it is written to PageCache, greatly increasing writing efficiency. Kafka also supports parameters that control whether data is returned on the PageCache or on the disk.
The advantage of Using PageCache in Kafka is that it does not consume JVM memory, and too much memory can also lead to increased GC time and frequent GC, which in turn affects Kafka’s throughput efficiency. No data will be lost when the JVM crashes, though data in PageCache will also be lost if the server crashes.
4. Batch operation
Kafka allows you to send and receive messages in batches to increase read and write efficiency
5. Data compression
Kafka supports receiving and sending compressed messages to increase network transmission efficiency. It is important to note that unpacking consumes CPU resources.