tKafkaOutput properties for Apache Spark Structured Streaming

Select the tKafkaConfiguration component to use for the Kafka connection settings.

Enter the name of the topic you want to publish messages to. This topic must already exist.

Enter the partition number to be used from the topic.

Enter the message key to assign to the Kafka records.

Select the Compress the data check box to compress the output data.

Enter the path to the directory where Spark stores the checkpoint data for this streaming query.

Checkpointing enables fault tolerance and allows a failed query to resume from where it stopped.

• Continuous: processes data continuously.
• Available now: processes all available data in a single batch, then stops the query.
• Fixed interval: runs the query at a fixed time interval. Enter the interval duration in the field that appears.

• Append: adds new rows without modifying existing rows.
• Update: writes only the rows that were updated since the last trigger.

Add the Kafka new producer properties you need to customize to this table.

For more information about the new producer properties you can define in this table, see the section describing the new producer configuration from the official Kafka documentation.

This component is used as an end component and requires an input link.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.