tFilterRow properties for Apache Spark Batch

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Logical operator used to combine conditions

Select a logical operator to combine simple conditions and to combine the filter results of both modes if any advanced conditions are defined.

And: returns the boolean value of true if all conditions are true; otherwise false. For each two conditions combined using a logical AND, the second condition is evaluated only if the first condition is evaluated to be true.

Or: returns the boolean value of true if any condition is true; otherwise false. For each two conditions combined using a logical OR, the second condition is evaluated only if the first condition is evaluated to be false.

Conditions

Click the plus button to add as many simple conditions as needed. Based on the logical operator selected, the conditions are evaluated one after the other in sequential order for each row. When evaluated, each condition returns the boolean value of true or false.

Input column: Select the column of the schema the function is to be operated on

Function: Select the function on the list

Operator: Select the operator to bind the input column with the value

Value: Type in the filtered value, between quotes if needed.

Use advanced mode

Select this check box when the operations you want to perform cannot be carried out through the standard functions offered, for example, different logical operations in the same component. In the text field, type in the regular expression as required.

If multiple advanced conditions are defined, use a logical operator between two conditions:

&& (logical AND): returns the boolean value of true if both conditions are true; otherwise false. The second condition is evaluated only if the first condition is evaluated to be true.

|| (logical OR): returns the boolean value of true if either condition is true; otherwise false. The second condition is evaluated only if the first condition is evaluated to be false.

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Batch component Palette it belongs to, appears only when you are creating a Spark Batch Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.