cKafka Standard properties

Broker List

Specify the list of Kafka message brokers in the form hostname1:port1,hostname2:port2,hostname3:port3.

Client Id

Specify an Id string of the client to pass to the server when making requests.

Topic

Type in a name for the message topic in the message broker.

GroupId

Enter the Id of the Kafka Connect cluster group.

The following options are available only when the cKafka is used as a Producer:

Partitioner

Enter the partitioner that determines how data is distributed across the Kafka cluster.

Serializer Class

Enter the class name of the serializer to be used.

Key Serializer Class

Enter the class name of the key serializer to be used.

Send Buffer (bytes)

The size of the TCP send buffer to use when sending data.

Request Required Acks

Specify whether the producer waits for an acknowledgement from the broker that the message was received by entering:

• 0, which means that the producer never waits for an acknowledgement from the broker;

• 1, which means that the producer gets an acknowledgement after the leader replica has received the data;

• -1 or all, which means that the producer gets an acknowledgement after all in-sync replicas have received the data.

Request Timeout (ms)

Specify the maximum amount of time in milliseconds that the client will wait for the response of a request. If the response is not received before the timeout elapses, the client will resend the request if necessary or fail the request if retries are exhausted.

Compression Codec

Select compression type from NONE, GZIP, SNAPPY, and LZ4.

Buffer Memory Size

Specify the total bytes of memory the producer can use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server, the producer will block for Max Block (ms) after which it will throw an exception.

Retries

Specify a value greater than 0 for the client to resend any record that failed to be sent with a potentially transient error.

Retry Backoff (ms)

Specify the amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

Batch Size

The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. This helps performance on both the client and the server. Specify the default batch size in bytes in this field.

Connection Idle Max (ms)

Specify the time in milliseconds after which idle connections will be closed.

Linger (ms)

Specify how long, in milliseconds, the producer must wait to group together any records that arrive into a single batched request, in order to reduce the number of requests. Note that, if the producer has received the amount of records specified by the Bath Size, it will send out the records immediately, regardless of what is specified in this setting.

Max Block (ms)

Specify the maximum amount of time in milliseconds that the producer will wait either because the buffer is full or metadata unavailable.

Max Request Size

Specify the maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests.

Receive Buffer (bytes)

Specify the size of the TCP receive buffer to use when reading data. If the value is -1, the OS default will be used.

Max in Flight Request

Specify the maximum number of unacknowledged requests that the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries if retries are enabled.

Metadata Max Age (ms)

Specify the period of time in milliseconds after which a refresh of metadata occurs even if there are no partition leadership changes to proactively discover any new brokers or partitions.

Reconnect Backoff (ms)

Specify the amount of time in milliseconds to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop.

For more information about Schema Registry, see the Confluent documentation.

This option is available when you have installed the 8.0.1-R2023-10 Talend Studio Monthly update or a later one delivered by Talend. For more information, check with your administrator.

The following options are available only when the cKafka is used as a Consumer:

Send Buffer (bytes)

Specify the size of the TCP send buffer to use when sending data. If the value is -1, the OS default will be used.

Retry Backoff (ms)

Specify the amount of time to wait before attempting to retry a failed request to a given topic partition. This avoids repeatedly sending requests in a tight loop under some failure scenarios.

Connection Idle Max (ms)

Specify the time in milliseconds after which idle connections will be closed.

Receive Buffer (bytes)

Specify the size of the TCP receive buffer to use when reading data. If the value is -1, the OS default will be used.

Metadata Max Age (ms)

Specify the period of time in milliseconds after which a refresh of metadata occurs even if there are no partition leadership changes to proactively discover any new brokers or partitions.

Reconnect Backoff (ms)

Specify the amount of time in milliseconds to wait before attempting to reconnect to a given host. This avoids repeatedly connecting to a host in a tight loop.

Auto Commit Enable

Select this check box to periodically commit the offset of messages in the background. Specify the frequency in milliseconds in the Auto Commit Interval (ms) that the consumer offsets are committed to Kafka.

Fetch Min (bytes)

Specify the minimum amount of data in bytes that the server should return for a fetch request. If insufficient data is available, the request will wait for that much data to accumulate before answering the request. The default setting of 1 byte means that fetch requests are answered as soon as a single byte of data is available or the fetch request times out waiting for data to arrive. Setting this to something greater than 1 will cause the server to wait for larger amounts of data to accumulate which can improve server throughput a bit at the cost of some additional latency.

Specify the maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by Fetch Min (bytes).

Auto Offset Reset

Choose what to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server from the following:

• EARLIEST: automatically reset the offset to the earliest offset;

• LATEST: automatically reset the offset to the latest offset;

• NONE: throw exception to the consumer if no previous offset is found for the consumer's group.

Heartbeat Interval (ms)

Specify the expected time in milliseconds between heartbeats to the consumer coordinator when using Kafka's group management facility. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. This value must be set lower than Session Timeout (ms), but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.

Maximum Partition Fetch (bytes)

Specify the maximum amount of data per-partition in bytes that the server will return. If the first message in the first non-empty partition of the fetch is larger than this limit, the message will still be returned to ensure that the consumer can make progress.

Session Timeout (ms)

Specify the timeout in milliseconds used to detect consumer failures when using Kafka's group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, the broker will remove this consumer from the group and initiate a rebalance.

Partition Assignor

Specify the class name of the partition assignment strategy that the client will use to distribute partition ownership amongst consumer instances when group management is used.

Request Timeout (ms)

Specify the maximum amount of time in milliseconds that the client will wait for the response of a request. If the response is not received before the timeout elapses the client will resend the request if necessary or fail the request if retries are exhausted.

For more information about Schema Registry, see the Confluent documentation.

This option is available when you have installed the 8.0.1-R2023-10 Talend Studio Monthly update or a later one delivered by Talend. For more information, check with your administrator.

Set the optional arguments in the corresponding table. Click [+] as many times as required to add arguments to the table. Then click the corresponding value field and enter a value. See the sitehttp://kafka.apache.org/documentation.html for available options.

SSL Key Password

Click [...] and enter the password of the private key in the key store file in double quotes.

SSL Keystore Location

Enter the location of the key store file.

SSL Keystore Password

Click [...] and enter the password for the key store file in double quotes. This is only needed if SSL Keystore Location is configured.

SSL Truststore Location

Enter the location of the trust store file.

SSL Truststore Password

Click [...] and enter the password for the trust store file in double quotes.

SSL Cipher Suits

Enter the list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported.

SSL Endpoint Algorithm

Enter the endpoint identification algorithm to validate server hostname using server certificate.

Kerberos Service Name

Enter the Kerberos principal name that Kafka runs as.

Security Protocol

Select the protocol to use to communicate with brokers from Plaintext, SSL, SASL over Plaintext, and SASL over SSL.

Usage rule

Limitation

n/a