tWindow properties for Apache Spark Streaming

Window duration

Enter, without quotation marks, the duration (in milliseconds) that defines the length of the window to be applied.

For example, if the batch size defined in the Spark configuration tab is 2 seconds, a window duration of 6 seconds means that 3 batches are handled each time this window is applied.

Define the slide duration

Select the Define the slide duration check box and in the field that is displayed, enter, without quotation marks, the time in milliseconds at the end of which the window is to be applied.

For example, if the batch size defined in the Spark configuration tab is 2 seconds, a slide duration of 4 seconds means the window is applied every 4 seconds; and if the window duration is 6 seconds, after two window applications there will be the overlap of one batch.

If you leave this check box clear, the slide duration is assumed to be the batch size defined in the Spark configuration tab.

Both the window duration and the slide duration must be multiples of the batch size defined in the Spark configuration tab.

Usage rule

This component is used as an intermediate step.

This component does not change the data schema but controls the pace of the processing of the micro-batches via the specific window.

This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.