tTopBy properties for Apache Spark Structured Streaming

A schema is a row description. It defines the fields (columns) processed by the component. When you create a Spark Job, avoid the reserved word line when naming fields.

• Built-In: create and store the schema locally for this component only.

• Repository: reuse a schema created and stored in the Repository across projects and Job designs.

Click Edit schema to modify the schema. If you modify a Repository schema, the available options include:

• View schema: view the schema without modifying it.

• Change to built-in property: change the schema to Built-in for local changes.

• Update repository connection: modify the Repository schema and choose whether to propagate the changes to other Jobs.

Enter the number of rows to be outputted. The current component selects this number of rows down from the first rows of the sorted data.

Select the column(s) from which the records are used as keys to group the input data.

You need to click [+] to add as many lines as required to use as keys for the grouping to be completed

• Click [+] to add as many lines as required for the sort to be completed.

• In the Schema column column, select the column from your schema, which the sort will be based on. Note that the order is essential as it determines the sorting priority.

• In the other columns, select how you need the data to be sorted. For example, if you need to sort the data in ascending alphabetical order (from A to Z), select alpha and asc in the corresponding columns.

This component is used as an intermediate step.

This component maintains a stateful ranking of the top N rows per partition group across micro-batches. If no partition key is defined, all rows are ranked together as a single global group.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.