tCollectAndCheck properties for Apache Spark Structured Streaming
Last updated: 9/30/2026These properties are used to configure tCollectAndCheck running in the Spark Structured Streaming Job framework.
The Spark Structured Streaming tCollectAndCheck component belongs to the Technical family.
The streaming version of this component is available in Talend Real-Time Big Data Platform and in Talend Data Fabric.
Basic settings
| Properties | Description |
|---|---|
| Configuration | Select this check box to retrieve connection information and credentials from
a configuration component. You must select this check box for the following types
of input data to be checked:
In the drop-down list that appears, select the configuration component from which Spark uses the configuration details to connect to the database. For example, to check Snowflake data, select the tSnowflakeConfiguration component. Information noteNote: If you want to retrieve data from S3, you do not have to use
tS3Configuration. Enter the full path of the file in the
Path or table name field instead.
|
| Type of input | Select the type of input data to check from the drop-down list. |
| Path or table name | Enter the file path or table name in double quotation marks. |
| Separator |
Enter a character, a string, or a regular expression to separate fields for the transferred data. |
| Line separator |
The separator used to identify the end of a row. |
| Use context variable |
If you have already created the context variable representing the reference file to be used, select this check box and enter this variable in the Variable name field that is displayed. The syntax to call a variable is context.VariableName. For more information about variables, see Using contexts and variables. |
| Reference data |
If you do not want to use context variables to represent the reference data to be used, enter this reference data directly in this field. |
| Keep the order from the reference |
If the RDDs to be checked are sorted, select this check box to keep your reference data ordered. |
| Output mode | Select the output mode from the drop-down list:
|
Usage
| Usage guidance | Description |
|---|---|
| Usage rule |
This component is used as an end component and requires an input link. This component is added automatically to a test case being created to show the test result in the console of the Run view. This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job. |
| Spark Connection |
You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job. This connection is effective on a per-Job basis. |