tFilterRow properties for Apache Spark Structured Streaming

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

• Built-In: You create and store the schema locally for this component only.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Click Edit schema to make changes to the schema. If you make changes, the schema automatically becomes built-in.

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

And: Returns true if all conditions are true; otherwise false. The second condition is evaluated only if the first is true.

Or: Returns true if any condition is true; otherwise false. The second condition is evaluated only if the first is false.

Input column: Select the schema column to apply the function to.

Function: Select the function from the list.

Operator: Select the operator to bind the input column with the value.

Value: Enter the filter value, in quotation marks if required.

If multiple advanced conditions are defined, use a logical operator to combine conditions:

&& (logical AND): Returns true if both conditions are true. The second condition is evaluated only if the first is true.

|| (logical OR): Returns true if either condition is true. The second condition is evaluated only if the first is false.

This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.