tS3Configuration properties for Apache Spark Structured Streaming

Enter the access key ID that uniquely identifies an AWS Account. For further information about how to get your Access Key and Secret Key, see Getting Your AWS Access Keys.

Enter the secret access key, constituting the security credentials in combination with the access Key.

To enter the secret key, click the [...] button next to the secret key field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Enter the location of the temp folder in S3. This folder will be automatically created if it has not existed by the time of the execution.

• Role ARN: enter the Amazon Resource Name of the role to assume.
• Role session name: enter a name to identify the assumed-role session.
• Session duration (minutes, 15mins~1hour): enter the duration of the session in minutes.
• External ID: enter the external ID if the trust policy of the role requires one.

Select this check box and select the region to connect to.

This setting appears only when Assume Role is enabled.

Select this check box and enter the Amazon S3 endpoint to use in the Endpoint field.

If you leave this check box clear, the default endpoint is used. This setting appears only when Assume Role is enabled.

Set region and Set endpoint

If the AWS administrator has enabled the STS endpoints for the regions you want to use for better response performance, select the Set region check box and then select the regional endpoint to be used.

If the endpoint you want to use is not available in this regional endpoint list, clear the Set region check box, then select the Set endpoint check box and enter the endpoint to be used.

This service allows you to request temporary, limited-privilege credentials for the AWS user authentication. Therefore, you still need to provide the access key and secret key to authenticate the AWS account to be used.

For a list of the STS endpoints you can use, see AWS Security Token Service. For further information about the STS temporary credentials, see Temporary Security Credentials. Both articles are from the AWS documentation.

This component is used with no need to be connected to other components.

Multiple tS3Configuration components are allowed per Job. The Assume Role option can be added to multiple tS3Configuration components.

You need to drop tS3Configuration along with the file system related subJob to be run in the same Job so that the configuration is used by the whole Job at runtime.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.