tLogRow properties for Apache Spark Structured Streaming

Schema and Edit schema

• A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

  Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

    If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

• Built-In: You create and store the schema locally for this component only.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

• Field Separator: Enter the separator to delimit data in the log display.
• Print header: Select this check box to include the header of the input flow in the output display.
• Print component unique name in front of each output row: Select this check box to show the unique name of the component in front of each output row, to differentiate outputs when several tLogRow components are used.
• Print schema column name in front of each value: Select this check box to retrieve column labels from the output schema.
• Use fixed length for values: Select this check box to use a fixed length for the value display.

• Print unique name: Shows the unique name of the component.
• Print label: Shows the label of the component.
• Print unique name and label: Shows both the unique name and the label.

• Append: Adds new rows to the output without modifying existing rows.
• Complete: Overwrites the entire output with the updated result after every trigger.
• Update: Writes only the rows that were updated since the last trigger.

• Available now: Processes all available data and stops.
• Fixed interval: Processes data at a fixed time interval. In the Duration field, enter the interval. For example, 10 seconds.

This component is used as an intermediate or an end step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.