A: the consumption side side parameters are defined in the class: org. Apache. Kafka. Clients. Consumer. ConsumerConfig.
1.1: Bootstrap. servers: Default: NULL list of host/port pairs used to establish the initial connection to the Kafka cluster. The client will use all the servers and not just the nodes configured here. Because these server addresses are only used to initiate connections and to discover all the members of the Kafka cluster from the existing configuration (the cluster changes at any time), this list does not need to include the full cluster address (but try to configure several to prevent the configured servers from going down) format: host1:port1,host2:port2,…
1.2: Client.dns.lookup: Default: Controls how the client uses DNS lookup. If configured as USE_ALL_DNS_IPS, connect to each returned IP address in turn until the connection is successfully established. If configured as resolve_canonical_bootstrap_servers_only, each boot address is resolved into a list of Canonical names.
1.3: Group. id: Default: NULL The unique identity of the consumer group to which the consumer belongs. In the following two cases, the group.id must be set: (1) Kafka-based offset management policy. (2) : KafKaconSumer uses Subscribe interface to subscribe messages.
Id: Default: NULL consumer instance id. Only non-empty strings are allowed. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any one time. If not set, the consumer will join the group as a dynamic member, which is traditional behavior.
1.5: session.timeout.ms: Default: 10000 milliseconds in the consumer group, detect the consumer failed session timeout time. The consumer will send the heartbeat message to the server in a fixed period. If the server does not receive the heartbeat message within a specified period, the consumer will be considered lost. At this point, the server kicks the node out of the consumer group and rebalances. Note: this value must be in the group. Min. Session. A timeout. Ms and group. The Max. Session. A timeout. Ms.
1.6: heartbeat. Interval. Ms: Default: 3000 ms expected heartbeat interval in the Kafka consumer group. The heartbeat is used to ensure that consumer meetings remain active and to promote rebalancing as new consumers join or leave the group. This value must be set to a configuration lower than session.timeout.ms. But it should usually be set to less than one-third of this value.
1.7: partition. The assignment. The strategy: default: RangeAssignor. Class when using the group management, client will use the class name of the partition strategy to assign ownership partition between consumer instance. By implementing org. Apache. Kafka. Clients. Consumer. ConsumerPartitionAssignor interface, you can insert custom allocation policy.
1.8: metadata.max.age.ms: Default: 5*60*1000 milliseconds to force the cycle time to refresh the metadata. You can proactively discover any new agents or partitions even if there are no changes in the partition leadership.
1.9:enable.auto.mit: Default: True If true, the consumer’s offset will be automatically committed in the background periodically.
1.10:auto.mit.interval. Ms: Default: 5000ms The frequency at which the consumer automatically submits offset. Valid when enable.auto.mit value is true.
1.11: Client. id: Default: “” The ID string to be passed to the server when a request is made. The purpose of this is to be able to trace request sources other than IP/port by allowing the logical application name to be included in the server-side request log.
1.12: Client. rack: Default: “” Rack identifier for this client. This can be any string value indicating the physical location of this client. This corresponds to the broker configuration “Broker.Rack”
1.13: Max. Partition. The fetch bytes: default: 1 * 1024 * 1024 (1 m). The maximum amount of data returned from each partition on the server per request. The consumer retrieves data from the server in bulk. If the number of the first non-empty partition retrieved is greater than this value, it will still be returned. The maximum amount of data returned on the server side is determined by the message.max.bytes configuration (this is the server configuration). Alternatively, configure the Settings via the max.message.bytes topic.
1.14: send.buffer.bytes: Default: 128*1024 The size of the TCP send buffer to use when sending data (SO_SNDBUF). If the value is -1, the OS default value is used.
1.15: receive.buffer.bytes: Default: 64*1024 The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default value is used.
1.16: Fetch.min.bytes: Default: 1 Gets the minimum amount of data returned by the request from the server. If not enough data is available, the request waits for a large amount of data to accumulate before answering the request. The default setting of 1 byte means that the extract request will get a response whenever one byte of data is available or the extract request timeout while waiting for the data to arrive. Setting this value to a value greater than 1 causes the server to wait for a larger amount of data to accumulate, which increases server throughput slightly but adds some latency.
1.17: Fetch.max.bytes: Default: 50*1024*1024 (50M) Maximum amount of data returned from the server per request. The consumer retrieves data from the server in bulk. If the number of the first non-empty partition retrieved is greater than this value, it will still be returned. The maximum amount of data returned on the server side is determined by the message.max.bytes configuration (this is the server configuration). Alternatively, configure the Settings via the max.message.bytes topic.
1.18: FETCH. MAX.WAIT. MS: Default: 500 milliseconds. If there is not enough data to satisfy the fetch.min.bytes request immediately, the maximum time that the server will block before responding to a FETCH request.
1.19: Reconnect. Backoff.ms: Default: 50 milliseconds The basic amount of time to wait before attempting to reconnect to a given host. This avoids repeated connections to the host in a tight loop. This fallback applies to all connection attempts from the client to the broker.
1.20: Reconnect.backoff.max.ms: Default: 1000ms Maximum time (milliseconds) to wait when reconnecting to an agent that repeatedly failed to connect
1.21: Retry. Backoff.ms: Default Value Value: 100 milliseconds Amount of time to wait before attempting to retry a failed request for a given theme partition. This prevents requests from being sent repeatedly in a tight loop under certain failure conditions.
1.22: auto. Offset. Reset: Default: latest. Optional values (“latest”, “earliest”, “none”) What if there is no initial offset in Kafka, or if the current offset no longer exists on the server (for example, because the data has been deleted) : Automatically resets the offset to the latest offset None: The consumer group did not find the positional offset and throws an exception
1.23: check. CRCS: Default: true Automatically checks CRC32 for consumed records. This ensures that no online or disk corruption of messages occurs. This check adds some overhead, so it may be disabled in cases where extreme performance is sought.
1.24: Metrics.sample.window.ms: Default: 30,000 milliseconds to calculate the time window for the measurement sample.
1.25: Metrics.num. Samples The number of samples reserved for calculation of the measurement.
1.26: Metrics. Recording. Level: Default: INFO. Optional values: DEBUG, INFO calculate the highest record level.
1.27: key. Deserializer: default: interface org.apache.kafka.com mon. Serialization. The deserializer implementation class. Deserialize the KEY.
1.28: the value. Deserializer: default: interface org.apache.kafka.com mon. Serialization. The realization of the deserializer class. Used to deserialize the VALUE VALUE.
1.29: request.timeout.ms: Default: 30,000ms Configuration controls the maximum time for a client to wait for an initiated request response. If no response is received before the timeout, a retry will be initiated before the number of retries is exhausted, and a failure will be obtained when the number of retries is exhausted.
1.30: Default.api.timeout. ms: Default: 60000 milliseconds The default timeout time that the consumer API may block. This default value is used when no timeout parameter is set.
1.31: Connection.max.idle.ms: Default: 9*60000 milliseconds to configure the maximum idle time for the connection. After this time the connection will be closed.
1.32: interceptor. Classes: default: empty list interface org. Apache. Kafka. Clients. Consumer. ConsumerInterceptor implementation class list. Use to add interceptors before the consumer receives the message.
1.33: max.poll.records: Default: 500 Maximum number of records retrieved in a single poll request.
1.34: max.poll.interval. Ms: Default: 300000 milliseconds Maximum delay time to call the poll command when using consumer group management. This value is the maximum idle time before the consumer retrieves the record. If the consumer does not call the poll command before this timeout is reached, the consumer is considered to have failed, and a rebalance of the consumer group is triggered to allocate the partition to other consumers. If the consumer’s group.instance.id configuration is non-empty, the partition will not be reassigned immediately after the timeout period is reached. The consumer will stop sending heartbeat messages and the partition will not be reassigned until the timeout period configured by session.timeout.ms is reached. This reflects the behavior of closed consumers.
1.35: exclude.internal. Topics: Default: true whether internal topics of the subscription mode should be excluded from the subscription topic
1.36: internal. Leave. Group. On. Close: default value: true consumers is turned off, if removed from the consumer groups
1.37: ISOLATE.LEVEL: Default: READ_UNCOMMITTED, optional: READ_COMMITTED, READ_UNCOMMITTED controls how transactional write messages are read. If READ_COMMITTED is configured, the consumer can only read transactional committed messages using the poll command. If you configured READ_UNCOMMITTED, you would get all the messages, even if the transaction broke. Non-transactional messages are returned unconditionally in either mode.
1.38: allow.auto.create.topics: Default: true Whether to allow automatic creation of topics on the server when subscribing or assigning topics. At the same time, the service side auto. Create. Switchable viewer. Must to true to automatically create the enable configuration.
1.39: security. Will: default: null interface org.apache.kafka.com mon. Security. The auth. SecurityProviderCreator implementation class, to implement security algorithm.
1.40: security.protocol: Default: PLAINTEXT and server communication protocol. Valid values are: utils.join (SecurityProtocol.names(), “, “)
2: on the production side parameters are defined in the class: org. Apache. Kafka. Clients. Producer. ProducerConfig
2.1: Bootstrap. servers: Default: NULL list of host/port pairs used to establish the initial connection to the Kafka cluster. The client will use all the servers and not just the nodes configured here. Because these server addresses are only used to initiate connections and to discover all the members of the Kafka cluster from the existing configuration (the cluster changes at any time), this list does not need to include the full cluster address (but try to configure several to prevent the configured servers from going down) format: host1:port1,host2:port2,…
2.2: Client.dns.lookup: Default: Controls how the client uses DNS lookup. If configured as USE_ALL_DNS_IPS, connect to each returned IP address in turn until the connection is successfully established. If configured as resolve_canonical_bootstrap_servers_only, each boot address is resolved into a list of Canonical names.
Memory: Default: 32*1024*1024L The total number of bytes of memory that producers can use to buffer records waiting to be sent to the server. If the record is sent faster than it can be transferred to the server, the production side will block the MAX_BLOCK_MS_CONFIG configuration for the time it needs, and then an exception will be thrown. This setting should correspond roughly to the total memory that will be used by the producer, but is not a hard limit. Because not all producers use buffers.
Retries: The default value is integer.max_value. Range: [0, Integer.max_value] Setting a value greater than zero will cause the client to resend any record that failed due to a temporary error. This retry is no different than the client resending the record when it receives an error. If MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION is set to something other than 1, then retry may change the order in which messages arrive at PARTITION. For example, if the first message fails to retry and the second message succeeds, the second message may reach Partition before the first. After configuring the timeout time of DELIVERY_TIMEOUT_MS_CONFIG, the timeout will fail even if the retry number has not been used up but the timeout has expired. In the same city case, you can leave this property unset and use DELIVERY_TIMEOUT_MS_CONFIG to control.
2.5: acks: Default value: “1”, optional value: “all”, “-1”, “0”, “1” The number of responses received by the Leader determines whether the producer request has been processed. If set to 0, the producer does not wait for a reply from the server and the message is immediately added to the socket buffer and is considered sent. In this case, there is no guarantee that the message has been sent to the server and the retries configuration will not work. If set to 1, the message is sent to the Leader and written to the Leader’s log without waiting for the follow reply. In such a case, the Leader can confirm that the message was received, but the Follow may have lost the message. If set to ALL, the Leader waits for all follow-ups to be answered before responding. This strongly guarantees that messages will not be lost. If set to -1, the effect is the same as if set to ALL.
2.6: Compression. Type: Default: NONE Compression type for all data generated by the producer. The default value is None, which means no compression. Valid values are: none, gzip, snappy, lz4, ZSTD. Compression is the compaction of a whole batch of data, so the effect of batch processing can also affect the compression ratio.
Whenever multiple messages are sent to the same partition, the producer will attempt to batch the records together to reduce the number of requests. This helps improve client and server performance. This configuration controls the default batch size in bytes. No attempt will be made to batch records larger than this size. The request sent to the broker will contain multiple batches, one for each partition. Smaller batch. sizes are not very generic and may reduce throughput. A very large batch.size might use a bit of memory waste because we always allocate a buffer of a specified batch size in anticipation of other records.
2.8: LINGER.MS: Default: 0 The producer combines all the records that arrive between request transmissions into a single batch request. Typically, this will only happen if the record arrives faster than it was sent. But in some cases, customers may want to reduce the number of requests even under the right load. This configuration sets a delay. Instead of sending the message immediately, the producer waits for a configured time to send the message in bulk, which is similar to the Nagle algorithm in TCP. When we configured BATCH_SIZE_CONFIG, linger.ms was the upper limit for waiting, even if the number of message bytes did not reach the configured value. If LINGER_MS_CONFIG is configured to 0, it does not wait.
2.9: delivery.timeout.ms: Default: 120*1000 milliseconds Maximum time to report success or failure after the send() method is called. This configuration limits the maximum time that messages can be sent late, wait for acknowledgement from the server, and fail to retry. When an unrecoverable error is encountered, the number of retries has been exhausted, and when the time limit has not been reached, the result will also be returned early. This value should not be less than the sum of REQUEST_TIMEOUT_MS_CONFIG and LINGER_MS_CONFIG.
2.10: Client.id: Default: “” The ID string to be passed to the server when the request is made. The purpose of this is to be able to trace request sources other than IP/port by allowing logical application names to be included in server-side request logging.
2.11: send.buffer.bytes: Default: 128*1024 The size of the TCP send buffer (SO_SNDBUF) to use when sending data If the value is -1, the OS default value is used.
2.12: receive.buffer.bytes: Default: 32*1024 The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default value is used.
2.13: max.request.size: Default: 1024*1024 Maximum size (bytes) of a request. This setting limits the maximum value when producers send messages in bulk, so as not to send a very large request. Note that the server also has a maximum value for the request, which may not be the same.
2.14: Reconnect. Backoff. ms: Default: 50L Basic amount of time to wait before attempting to reconnect to a given host. This avoids repeated connections to the host in a tight loop.
2.15: RECONNECT.BACKOFF.MAX.MS: Default: 1000L maximum wait time to reconnect to the server. If provided, the reconnection per host will multiply with each successive connection failure up to this maximum.
2.16: Retry. Backoff. ms: Default: 100L Amount of time to wait before attempting to retry a failed request for a given theme partition. This prevents requests from being sent repeatedly in a tight loop under certain failure conditions.
2.17: Max. Block. Ms: default: 60 * 1000 control KafkaProducer. The send () and KafkaProducer partitionsFor () command block of time. These methods may be blocked because the buffer is full or metadata is unavailable. Blocks in user-supplied serializers or partitioners do not count towards this timeout.
2.18: request.timeout.ms: Default: 30*1000 Configuration controls the maximum time for a client to wait for a response to a request. If no response is received before the timeout is reached, it will be retried or it will fail. The value of this configuration should be greater than the value of replica.lag.time.max.ms to reduce the possibility of message duplication due to unnecessary producer retries.
2.19: metadata.max.age.ms: Default: 5*60*1000 A period of time in milliseconds after which we force a refresh of the metadata so that we can actively discover any new agents or partitions even if we do not see any partition leadership changes.
2.20: Metrics.sample.window.ms: Default: 30000 to calculate the time window for the measurement sample.
2.21: Metrics.num. Samples: Default: 2 The number of samples reserved for calculating the measure.
2.22: Metrics. Recording. Level: Default: INFO. Optional value: INFO, the highest recorded level for the DEBUG metric.
2.23: Metric.reporters: Default: Null list of classes used as metric reporters. Is org.apache.kafka.com mon. Metrics. MetricsReporter interface implementation class. JMXReporter is always included in the registration JMX statistics.
2.24: Max. In. Flight. Requests. Per. Connection: default: 5 before the block, the client will be on a single connection to send one of the biggest did not confirm the number of requests. Note that if this setting is set to greater than 1 and there is a failed send, there is a risk that the message will be reordered due to retry.
2.25: key. Serializer: default: no interface org.apache.kafka.com mon. Serialization. The realization of the serializer class, for the serialization of key.
2.26: value. Serializer: default: no interface org.apache.kafka.com mon. Serialization. The realization of the serializer class, for the serialization of the value.
2.27: Connection.max.idle.ms: Default: 9*60*1000 Close idle connections after the number of milliseconds specified in this configuration.
2.28: the partitioner. Class: default: no interface org. Apache. Kafka. Clients. The producer. The realization of the partitioner class, used to customize the subdivision algorithm.
2.29: interceptor. Classes: default: empty interface org. Apache. Kafka. Clients. Producer. ProducerInterceptor implementation class list of interceptors. Allows messages to be intercepted before they are sent to the Kafka cluster. By default, there are no interceptors.
2.30: security.protocol: Default: The communication protocol between PLAINTEXT and the server. Possible values are: utils.join (SecurityProtocol.names(), “, “)
2.31: security. Will: default: empty interface org.apache.kafka.com mon. Security. The auth. SecurityProviderCreator implementation class list, to implement security algorithm.
2.32: enable. iDempotence: Default: false When set to “true”, the producer will ensure that only one copy of each message is written to the stream. If “false”, then a producer retry due to server-side failure, etc., may write a copy of the retry message to the stream. Note that if set to true, then the value of the configuration MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION must be less than or equal to 5, the RETRIES_CONFIG configuration must be greater than 0, and the ACKS_CONFIG configuration must be ALL. If the user does not explicitly set these values, the appropriate values are selected. If incompatible values are set, a ConconfigException is thrown.
2.33: transaction.timeout.ms: Default: 60000 Maximum time (milliseconds) that the transaction coordinator waits for a producer to update the state of a transaction before actively aborting an ongoing transaction. If this value is greater than the agent in the transaction. Max. A timeout. Ms Settings, will fail, and InvalidTransactionTimeout mistakes.
Transaction. id: Default: No transactionalID for transactional passing. This supports reliability semantics across multiple producer sessions because it allows the client to guarantee that a transaction using the same TransactionAlid has completed before starting any new transactions. If no TransactionalID is provided, the producer is limited to idempotent transmission. Note that if TransactionAlid is configured, the enable. iDempotence configuration must be true. The default value is null, which means that transactions cannot be used.