Configuring and running your Spark Job with CDP Public Cloud Data Hub on AWS
Talend Studio allows you to deploy and execute your Spark Streaming and Spark Batch Jobs on a remote JobServer with a CDP Public Cloud Data Hub on AWS instance.
Before you begin
- The JobServer settings are defined correctly in Talend Studio to run your Job remotely. For more information see, Configuring remote execution (Talend > Run/Debug).
- The AWS instance environment is defined in Cloudera Management Console. For more information, see Register an AWS environment from the official Cloudera documentation.
- The cluster on AWS is defined in the Cloudera Management Console. For more information, see Create a custom cluster on AWS from the official Cloudera documentation.
Procedure
-
Connect to your Cloudera Management console and go to the Data Hub
Clusters tab and then Hardware tab.
-
Connect to your AWS Management Console and from the VPC Management
Console, make sure that the ports in the Inbound
rules and Outbound rules tabs that you set up
for the JobServer are open.
-
Connect to Cloudera Manager and from the
Clusters tab, download all the configuration files from
your cluster and unzip them all in the same path on your local machine.
-
Connect to Talend Studio and
set up manually the Hadoop connection using the Import configuration from
local files option. For more information, see the third step in Setting up the Hadoop connection.
Information noteNote:- You do not have to select any Cloudera version in the drop-down list. As Talend Studio uses the configuration files from the CDP Public Cloud instance clusters, it will use the runtime version defined in it.
- You must enable SSL and Kerberos.