tAvroOutput properties for Apache Spark Structured Streaming

Select the configuration component to be used to provide the configuration information for the connection to the target file system such as HDFS.

If you leave this check box clear, the target file system is the local system.

The configuration component to be used must be present in the same Job. For example, if you have dropped a tHDFSConfiguration component in the Job, you can select it to write the result in a given HDFS system.

Property Type

Select the way the file path and the schema will be set.

• Built-In: The file path and the schema will be set locally for this component.

• Repository: The file details stored centrally in Repository > Metadata will be reused by this component.

  You need to click the [...] button next to it and in the pop-up Repository Content dialog box, select the file to be reused, and all related properties will be automatically filled in.

A schema is a row description. It defines the fields (columns) processed by the component. When you create a Spark Job, avoid the reserved word line when naming fields.

• Built-In: create and store the schema locally for this component only.

• Repository: reuse a schema created and stored in the Repository across projects and Job designs.

Click Edit schema to modify the schema. If you modify a Repository schema, the available options include:

• View schema: view the schema without modifying it.

• Change to built-in property: change the schema to Built-in for local changes.

• Update repository connection: modify the Repository schema and choose whether to propagate the changes to other Jobs.

To write to cloud storage, add the corresponding configuration component to the Job. For example, use tS3Configuration for Amazon S3, tGSConfiguration for Google Cloud Storage, or tAzureFSConfiguration for Azure Data Lake Storage.

To use a cloud storage path, add the corresponding configuration component to the Job. For example, use tS3Configuration for Amazon S3, tGSConfiguration for Google Cloud Storage, or tAzureFSConfiguration for Azure Data Lake Storage.

• Append: adds data to an existing directory, or creates a new directory if none exists.
• Overwrite: clears the existing data in the Folder directory, then creates the output files and writes data to them. The checkpoint location is not cleared.

• Available now: processes all available data in a single batch, then stops the query.
• Fixed interval: runs the query at a fixed time interval. Enter the interval duration in the field that appears, for example 10 seconds.

This component is used as an end component and requires an input link.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.