tAggregateRow properties for Apache Spark Structured Streaming

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

• Built-In: You create and store the schema locally for this component only.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Click Edit schema to make changes to the schema. If you make changes, the schema automatically becomes built-in.

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Output Column: Select the column label from the list based on the schema structure you defined. Add as many output columns as needed to refine the aggregation. For example, select Country to calculate an average for each country, or select Country and Region to compare regions across countries.

Input Column: Match the input column label to the output column when the aggregation set needs a different output label.

Output Column: Select the destination field from the list.

Function: Select the operator among:

Input column: Select the input column whose values are aggregated.

Ignore null values: Select the check boxes for the columns where NULL values should be ignored.

This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.