Skip to main content Skip to complementary content

tKafkaCommit properties for Apache Spark Streaming

These properties are used to configure tKafkaCommit running in the Spark Streaming Job framework.

The Spark Streaming tKafkaCommit component belongs to the Messaging family.

This component is available in Talend Real-Time Big Data Platform and Talend Data Fabric.

Information noteImportant: For Spark Streaming Jobs, Talend Studio does not support a specific version of Kafka but relies on a Kafka broker version compatibility provided by Spark. The Kafka broker version supported depends on the Spark version you use. For each Spark version, Talend Studio supports the targeted Kafka broker version provided by Spark. As of today Talend Studio relies on Spark compatibility statement, and therefore supports Kafka broker version 0.10.0 onwards. For more information, see Spark Streaming + Kafka Integration Guide on the official Spark documentation.

Basic settings

Commit offsets from

Select the tKafkaInput component from which the consumed messages are committed.

Usage

Usage rule

To use tKafkaCommit, the Auto-commit offests check box in the Basic settings view of the tKafkaInput component must be cleared.

This component is supported with Spark v2.0 and onwards in the Local Spark mode. It is typically used standalone to commit a batch of offset.

This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!