background

When I was retrieving messages, I accidentally found some messages with a time value of -1, as follows:

​

The other two messages had a normal time.

To find out why, I looked up kafka’s official documentation and came up with a conclusion.

The official instructions

There are several versions of the kafka message format. In older versions, there was no timestamp field.

Official document address: Apache Kafka

Prior to 0.11, messages were sent and stored in a collection of messages, with metadata information for each message. At this point, messages were formatted in two versions: version 0 and version 1, as follows:

​

In the version 1 message format, the message time information is available.

The above message format was before version 0.11. What about version 0.11 and later? Here are the official notes:

​

Even the name has changed since 0.11. Before 0.11, messages were called message, and the batches sent were called Message sets. The Batch sent from 0.11 is called record and the Batch sent is called Record Batch. But the meaning is still the same.

In 0.11, the message Header was introduced: the Record Header.

The message version is called V2.

From the V2 message format shown in the documentation, you can see that general information, such as compression algorithms, is missing.

This is because this version has been optimized to extract the general information and put it in the outer message set instead of saving every message, so you can see that the Arrtibutes section is now unused. The delta followed in the message format above is the relative value, a relative value calculated in the first clause of the current batch.

The source code that

I checked the lot code: kafka/ProducerRecord Java at 0.10.0, apache/kafka, dead simple

The timestamp field was available in the 0.10 release when Kafka sent messages, but not in the previous release.

I’m doing an example here.

The ProdcuerRecord properties for version 0.9 are as follows:

    private final String topic;
    private final Integer partition;
    private final K key;
    private final V value;
Copy the code

Version 0.10 has timestamp:

    private final String topic;
    private final Integer partition;
    private final K key;
    private final V value;
    private final Long timestamp;
Copy the code

Version 0.11 follows, introducing headers:

    private final String topic;
    private final Integer partition;
    private final Headers headers;
    private final K key;
    private final V value;
    private final Long timestamp;
Copy the code

This is still true with the latest 3.1 release.

In addition, the default for this time field is the creation time set by the producer, which is the system time of the client at that time. The time type can be modified as follows: the default is broker and a property with a value of CreateTime:

​

You can also change this value to LogAppendTime, which is the system time at which the broker side writes.

conclusion

If the time field of the message is -1, the client version is too old and the message format used does not support the time field.

note

Github-xxd763795151 / Kafka-console-UI: a lightweight visual management platform for Kafka that is easy to use