# tS3Configuration properties for Apache Spark Batch These properties are used to configure tS3Configuration running in the Spark Batch Job framework. The Spark Batch tS3Configuration component belongs to the Storage family. The component in this framework is available in all subscription-based Talend products with Big Data and Talend Data Fabric. ## Basic settings | Properties | Description | | --- | --- | | Access Key | Enter the access
key ID that uniquely identifies an AWS Account. For further information about how to
get your Access Key and Secret Key, see [Getting Your AWS Access
Keys](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/get-aws-keys.html). | | Secret Key | Enter the secret access key,
constituting the security credentials in combination with the access Key.

To enter the secret key, click the [...]
button next to the secret key field, and then in the pop-up dialog box enter the password
between double quotes and click OK to save the
settings. | | Hadoop S3 version | Select the Hadoop version you are using from the drop-down list.

If the
cluster has S3 libraries, select Use cluster's libraries
to avoid conflicts by uploading non aligned dependencies. If the cluster does
not have S3 libraries, select the version that matches the Hadoop libraries on
the cluster.

This parameter is not available when the Spark Job is
running on Spark Universal in Databricks,
Local, or Standalone
modes. | | File System | Select the file system type from the drop-down list:

S3A File System to use the standard Hadoop connector for accessing Amazon S3.
EMR File System to leverage Amazon EMR's connector with features like Consistent View and advanced AWS integration.

This field is only available when you have selected the Inherit
credentials from AWS check box. | | Use EMRFS consistent view | Select this check box to use the EMR File System (EMRFS) consistent view.
This option allows EMR clusters to check for list and read-after-write consistency
for Amazon S3 objects that are written by or synced with EMRFS.

Information noteNote: Avoid
alternatively switching the consistent view on and off on a single bucket, as
it might create inconsistency errors. If this issue occurs, you can fix the
inconsistencies using the sync command in the EMRFS CLI. For
more information, see [EMRFS CLI Reference](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emrfs-cli-reference.html).

This feature is
available when you are using the Amazon EMR 5.29 distribution. | | EMRFS metadata table | Enter the name of the metadata DynamoDB table to be used.

Information noteNote: The
default metadata table name is
EmrFSMetadata.

This field is only available
when you have selected the Use EMRFS
consistent view check box. | | Bucket name | Enter the bucket name and its folder you need to use. You
need to separate the bucket name and the folder name using a slash (/). | | Temp folder | Enter the location of the temp folder in S3.
This folder will be automatically created if it has not existed by the time of the
execution. | | Inherit credentials from AWS | Select this check box to obtain AWS security credentials from your IAM role.
This option is available for Amazon EMR and Databricks on AWS clusters. To use
this option, the cluster must be started and your Job must be running on this
cluster. For more information, see [Using an IAM Role to Grant Permissions to Applications
Running on Amazon EC2 Instances](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
.

This option enables you to develop your Job without having to put any
AWS keys in the Job, thus easily comply with the security policy of your
organization. | | Use SSE encryption | Select this check box
to use the SSE-KMS encryption service enabled on AWS to read or write the
encrypted data on S3.

On the EMR side, the SSE-KMS service must have been enabled with the
Default encryption feature and a customer managed CMK
specified for the encryption.

For further information
about the AWS SSE-KMS encryption, see [Protecting Data Using Server-Side Encryption](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html) from
the AWS documentation.

For further information about how to enable the Default
Encryption feature for an Amazon S3 bucket, see [Default encryption](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/default-bucket-encryption.html) from the AWS documentation.

This property is available only when you are using Amazon EMR
distributions. | | Bucket encryption | Select the default encryption you used for your bucket from the drop-down list:

aws:kms: server side encryption with AWS KMS-Managed Keys (SSE-KMS)
SSE-S3: server side encryption with Amazon S3-Managed Keys (SSE-S3)

This property is available only when you are using Amazon EMR
distributions. | | Use 'in-transit' encryption | Select this check box to enable the encryption of data in transit.

Information noteNote: The in-transit encryption is selected by default. If you deactivate this
option, you do not have to set up a KMS encrypted EMR cluster.

This property is available only when you are using Amazon EMR
distributions with SSE-KMS encryption. | | Assume role | Select this check box to make your Job temporarily
assume a role and the permissions associated with this role.

Ensure that access to this
role has been granted to your user account by the trust policy associated to
this role. If you are not certain about this, ask the owner of this role or
your AWS administrator.

After selecting this check box, specify the parameters the administrator of the
AWS system to be used defined for this role.

ARN: the Amazon Resource Name (ARN) of the role to assume. You can find this ARN name on the Summary page of the role to be used on your AWS portal, for example, this role ARN could read like am:aws:iam::[aws_account_number]:role/[role_name].
Role session name: enter the name you want to use to uniquely identify your assumed role session. This name can contain upper- and lower-case alphanumeric characters with no spaces. You can also include underscores or any of the following characters: =,.@-.
Session duration (minutes): the duration (in minutes) for which you want the assumed role session to be active. This duration cannot exceed the maximum duration which your AWS administrator has set. The duration defaults to 3600 seconds if you give it no value.

The External
ID parameter is required only if your AWS administrator
or the owner of this role has defined an external ID when they set up a
trust policy for this role.

Policy: enter an IAM policy in JSON format that you want to use as a session policy. Use session policies to limit the permissions of the session. The resulting session's permissions are the intersection of the role's identity-based policy and the session policies.
Policy ARNs: enter the Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. Use managed session policies to limit the permissions of the session. The policies must exist in the same account as the role. The resulting session's permissions are the intersection of the role's identity-based policy and the session policies.
Serial Number: when you assume a role, the trust policy of this role might require Multi-Factor Authentication (MFA). In this case, you must indicate the identification number of the hardware or virtual MFA device that is associated with the user who assumes the role.
Tags: list session tags in the form of key-value pairs. You can then use these session tags in policies to allow or deny access to requests.
Token Code: when you assume a role, the trust policy of this role might require Multi-Factor Authentication (MFA). In this case, you must indicate a token code. This token code is a time-based one-time password produced by the MFA device.
Transitive Tag Keys: list session tags in the form of key-value pairs that you want to persist to the next role in a role chain.

In addition, if the AWS administrator has enabled the
STS endpoints for given regions you want to use for better response
performance, use the Set STS region check box or the
Set STS endpoint check box in the
Advanced settings tab.

This check box is available only for the following distributions Talend supports:

CDH 5.10 and onwards (including the dynamic support for the latest Cloudera distributions)
HDP 2.5 and onwards
EMR 5.15 and onwards
CDP Private Cloud Base 7.1

This check box is also available when you are using
Spark V1.6 and onwards in the Local Spark mode in the Spark configuration tab. | | KMS key id | Enter the ID of the customer managed CMK you want to use for the
encryption.

Information noteNote: You can either use one of the following format:

arn:aws:kms:region:account-id:key/key-id
key-id

The KMS key id field is only available when you select
the Use SSE encryption check box. This feature is
available when you are using Amazon EMR distribution with Spark. | | Set region | Select this check box and select the region
to connect to.

This feature is available when you are
using one of the following distributions with Spark:

Amazon EMR V4.5 and up to V5.15. EMR uses EMRFS from V5.29
MapR V5.0 and onwards
Hortonworks Data Platform V2.4 and onwards
Cloudera V5.8 and onwards. For Cloudera V5.8, the Spark version must be 2.0.
Cloudera Altus

| | Set endpoint | Select this
check box and in the Endpoint
field that is displayed, enter the Amazon region endpoint you need to use. For a
list of the available endpoints, see [Regions and Endpoints](http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region).

If you leave this check box clear, the endpoint will be the default
one defined by your Hadoop distribution, while this check box is not available when
you have selected the Set
region check box and in this situation the value selected from the
Set region list is
used.

This feature is available when you are
using one of the following distributions with Spark:

Amazon EMR V4.5 and up to V5.15. EMR uses EMRFS from V5.29
MapR V5.0 and onwards
Hortonworks Data Platform V2.4 and onwards
Cloudera V5.8 and onwards. For Cloudera V5.8, the Spark version must be 2.0.
Cloudera Altus

| ## Advanced settings | Properties | Description | | --- | --- | | Set region and Set endpoint | If the AWS
administrator has enabled the STS endpoints for the regions you want to use for
better response performance, select the Set
region check box and then select the regional endpoint to be
used.

If the endpoint you
want to use is not available in this regional endpoint list, clear the
Set
region check box, then select the Set
endpoint check box and enter the endpoint to be used.

This service allows you to request
temporary, limited-privilege credentials for the AWS user authentication.
Therefore, you still need to provide the access key and secret key to
authenticate the AWS account to be used.

For a list of the STS endpoints you can
use, see [AWS Security Token Service](https://docs.aws.amazon.com/general/latest/gr/rande.html#sts_region). For
further information about the STS temporary credentials, see [Temporary Security Credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html).
Both articles are from the AWS documentation. | ## Usage | Usage guidance | Description | | --- | --- | | **Usage rule** | This component is used with no need to be connected to other
components.

Multiple tS3Configuration components are allowed per Job. The Assume
Role option can be added to multiple tS3Configuration
components.

You need to drop tS3Configuration along with the file system related
subJob to be run in the same Job so that the configuration is used by the whole Job
at runtime. | | Spark Connection | In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

Yarn mode (Yarn client or Yarn cluster):
- When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab.
- When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab.
- When using Altus, specify the S3 bucket or the Azure Data Lake Storage for Job deployment in the Spark configuration tab.
- When using on-premises distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration Apache Spark Batch or tS3Configuration Apache Spark Batch.

If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem).

This connection is effective on a per-Job basis. | | Limitation | Due to license incompatibility, one or more JARs required to use this component
are not provided. You can install the missing JARs for this particular component by
clicking the Install button on the Component tab view. You can also find out and add all missing
JARs easily on the Modules tab in the Integration perspective of Talend Studio. For
details, see [Installing external modules](https://help.qlik.com/talend/r/en-US/8.0/studio-user-guide/installing-external-modules). |