tMap properties for Apache Spark Structured Streaming

When you click the Property Settings button at the top of the input area, a Property Settings dialog box is displayed in which you can set the following parameters:

• If you do not want to handle execution errors, select the Die on error check box (selected by default). It will kill the Job if there is an error.

• To maximize the data transformation performance in a Job that handles multiple lookup input flows with large amounts of data, you can select the Lookup in parallel check box.

• Temp data directory path: enter the path where you want to store the temporary data generated for lookup loading. For more information on this folder, see Solving memory limitation issues in tMap use.

• Max buffer size (nb of rows): enter the size of physical memory, in number of rows, you want to allocate to processed data.

• Auto: the default setting displays links as curves.
• Curves: the mapping displays as curves.
• Lines: the mapping displays as straight lines. The Lines option can slightly improve performance.

Ensure that all lookup tables fit in memory.

This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.