# tKafkaInput properties for Apache Spark Structured Streaming

[Preview](https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/Introduction/feature-preview-program-overview.htm)

These properties are used to configure tKafkaInput running in the Spark Structured Streaming Job framework.

The Spark Structured Streaming tKafkaInput component belongs to the Messaging family.

The streaming version of this component is available in Talend Real-Time Big Data Platform and in Talend Data Fabric.

## Basic settings

| Properties | Description |
| --- | --- |
| Schema and Edit schema | A schema is a row description. It defines the number of fields<br>(columns) to be processed and passed on to the next component. When you create a Spark<br>Job, avoid the reserved word line when naming the<br>fields.<br><br><ul> <li> <p> <span>Built-In</span>: You create and store the schema locally for this component                only.</p> </li> <li> <p> <span>Repository</span>: You have already created the schema and stored it in the                Repository. You can reuse it in various projects and Job designs.</p> </li> </ul><br><br>Click Edit schema to make changes to<br>the schema. If you make changes, the schema automatically becomes built-in.<br><br><ul> <li> <p> <span>View schema</span>: choose this                option to view the schema only.</p> </li> <li> <p> <span>Change to built-in property</span>:                choose this option to change the schema to <span>Built-in</span> for local changes.</p> </li> <li> <p> <span>Update repository connection</span>:                choose this option to change the schema stored in the repository and decide whether                to propagate the changes to all the Jobs upon completion.</p> <p>If you just want to                propagate the changes to the current Job, you can select <span>No</span> upon completion and choose this schema metadata                again in the <span>Repository Content</span>                window.</p> </li> </ul> |
| Configuration component | Select the tKafkaConfiguration component to use for<br>the Kafka connection settings. |
| Starting offset | Select the starting point from which the messages of a topic are consumed.<br><br>In Kafka, the sequential ID number of a message is called *offset*. From this list, you can select From<br>beginning to start consumption from the oldest message of the entire topic,<br>or select From latest to start from the latest message that<br>has been consumed by the same consumer group and of which the offset is tracked by Spark<br>within Spark checkpoints.<br><br>Each consumer group has its own counter to remember the position of a message it has<br>consumed. For this reason, once a consumer group starts to consume messages of a given<br>topic, a consumer group recognizes the latest message only with regard to the position where<br>this group stops the consumption, rather than to the entire topic. Based on this principle,<br>the following behaviors can be expected:<br><br><ul> <li> <p>A topic has for example <em>100</em> messages. If a                consumer group has stopped the consumption at the message of the offset <em>50</em>, then when you select <span>From                    latest</span>, the same consumer group restarts from the offset <em>51</em>.</p> </li> <li> <p>If you create a new consumer group or reset an existing consumer group, which, in                either case, means this group has not consumed any message of this topic, then when                you start it from latest, this new group starts and waits for the offset <em>101</em>.</p> </li> </ul> |
| Topic name | Enter the name of the topic from which tKafkaInput<br>receives the feed of messages. |
| Group ID | Enter the name of the consumer group to which you want the current consumer (the tKafkaInput component) to belong.<br><br>This consumer group will be created at runtime if it does not exist at that moment.<br><br>This property is available only when you are using Spark 2.0 or the Hadoop distribution to<br>be used is running Spark 2.0. If you do not know the Spark version you are using, ask the<br>administrator of your cluster for details. |
| Set number of records per second to read from each Kafka<br>partition | Select this check box to set the maximum number of records read from each<br>Kafka partition per trigger. In the field that appears, enter the maximum number<br>of records. |
| Enable watermarking | Select this check box to activate watermarking and select the time mode to<br>be used:<br><br><ul> <li><span>Event time</span>: Select to use a timestamp column                     representing when each event occurred, then select the column in the                        <span>Watermark column (timestamp)</span> drop-down list.</li> <li><span>Processing time</span>: Select to use the time when Spark                     reads each record.</li> </ul><br><br>In the Watermark delay field, specify how long Spark<br>waits for late data before finalizing each window. |

## Advanced settings

| Properties | Description |
| --- | --- |
| Do not fail on data loss | Select this check box to allow the streaming query to continue processing<br>when Kafka offsets are missing — for example, because topics were deleted or<br>offsets are out of range. If cleared, the query stops with an error. |
| Kafka properties | Add the Kafka consumer properties you need to customize to this table. For example, you<br>can set a specific *zookeeper.connection.timeout.ms* value<br>to avoid *ZkTimeoutException*.<br><br>For further information about the consumer properties you can define in this table, see<br>the section describing the consumer configuration in Kafka's documentation in http://kafka.apache.org/documentation.html#consumerconfigs. |
| Use hierarchical mode | Select this check box to map the binary (including hierarchical) Avro schema to the<br>flat schema defined in the schema editor of the current component. If the Avro<br>message to be processed is flat, leave this check box clear.<br><br>Once selecting it, you need set the following parameter(s):<br><br><ul> <li> <p> <span>Local path to the avro                    schema</span>: browse to the file which defines the                schema of the Avro data to be processed.</p> </li> <li> <p> <span>Mapping</span>: create the map                between the schema columns of the current component and the data stored                in the hierarchical Avro message to be handled. In the                <span>Node</span> column, you need to                enter the JSON path pointing to the data to be read from the                Avro message.</p> </li> </ul> |

## Usage

| Usage guidance | Description |
| --- | --- |
| **Usage rule** | This component is used as a start component and requires an output link.<br><br>This component, along with the Spark Structured Streaming component Palette it belongs to,<br>appears only when you are creating a Spark Structured Streaming Job. |
| Spark Connection | You need to use the Structured Streaming Configuration<br>tab in the Run view to define the connection to a Spark cluster for<br>the whole Job.<br><br>This connection is effective on a per-Job basis. |