If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the MapR ticket authentication configuration in addition or as an alternative by following the explanation in Connecting to a security-enabled MapR.
Keep in mind that this configuration generates a new MapR security ticket for the username defined in the Job in each execution. If you need to reuse an existing ticket issued for the same username, leave both the Force MapR ticket authentication check box and the Use Kerberos authentication check box clear, and then MapR should be able to automatically find that ticket on the fly.
Hive principal uses the value of hive.metastore.kerberos.principal. This is the service principal of the Hive Metastore.
HiveServer2 local user principal uses the value of hive.server2.authentication.kerberos.principal.
HiveServer2 local user keytab uses the value of hive.server2.authentication.kerberos.keytab
Metastore URL uses the value of javax.jdo.option.ConnectionURL. This is the JDBC connection string to the Hive Metastore.
Driver class uses the value of javax.jdo.option.ConnectionDriverName. This is the name of the driver for the JDBC connection.
Username uses the value of javax.jdo.option.ConnectionUserName. This, as well as the Password parameter, is the user credential for connecting to the Hive Metastore.
Password uses the value of javax.jdo.option.ConnectionPassword.
In the Trust store path field, enter the path, or browse to the TrustStore file to be used. By default, the supported TrustStore types are JKS and PKCS 12.
To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.
Hortonworks Data Platform 2.0 +
Cloudera CDH4 +
Pivotal HD 2.0 +
Amazon EMR 4.0.0 +
Select the Set resourcemanager scheduler address check box and enter the Scheduler address in the field that appears.
Select the Set jobhistory address check box and enter the location of the JobHistory server of the Hadoop cluster to be used. This allows the metrics information of the current Job to be stored in that JobHistory server.
Select the Set staging directory check box and enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution.
Allocate proper memory volumes to the Map and the Reduce computations and the ApplicationMaster of YARN by selecting the Set memory check box in the Advanced settings view.
Select the Set Hadoop user check box and enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed.
Select the Use datanode hostname check box to allow the Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname property to true. When connecting to a S3N filesystem, you must select this check box.
In the parent level, register the database connection to be shared in the Basic settings view of the connection component which creates that very database connection.
In the child level, use a dedicated connection component to read that registered database connection.
If available in this Distribution drop-down list, the Microsoft HD Insight option allows you to use a Microsoft HD Insight cluster. For this purpose, you need to configure the connections to the HD Insight cluster and the Windows Azure Storage service of that cluster in the areas that are displayed. For detailed explanation about these parameters, search for configuring the connection manually on Talend Help Center (https://help.talend.com).
If you select Amazon EMR, find more details about Amazon EMR getting started in Talend Help Center (https://help.talend.com).
The Custom option allows you to connect to a cluster different from any of the distributions given in this list, that is to say, to connect to a cluster not officially supported by Talend .
Select Import from existing version to import an officially supported distribution as base and then add other required jar files which the base distribution does not provide.
Select Import from zip to import the configuration zip for the custom distribution to be used. This zip file should contain the libraries of the different Hadoop elements and the index file of these libraries.
Note that custom versions are not officially supported by Talend . Talend and its community provide you with the opportunity to connect to custom versions from the Studio but cannot guarantee that the configuration of whichever version you choose will be easy, due to the wide range of different Hadoop distributions and versions that are available. As such, you should only attempt to set up such a connection if you have sufficient Hadoop experience to handle any issues on your own.
In this dialog box, the active check box must be kept selected so as to import the jar files pertinent to the connection to be created between the custom distribution and this component.
For a step-by-step example about how to connect to a custom distribution and share this connection, see Hortonworks.
When you select LOAD, you are moving or copying data from a directory you specify.
When you select INSERT, you are moving or copying data based on queries.
Custom: this option allows you connect to a distribution supporting Tez but not officially supported by Talend .
If you select Table as destination, you can still choose to append data to or overwrite the contents in the specified table.
If you select Directory as destination, you are overwriting the contents in the specified directory
If you have selected LOAD: this is the path to the data you want to copy or move into the specified Hive table.
If you have selected INSERT: this is the directory to which you want to export data from a Hive table. With this action, the File path field is available only when you have selected Directory from the Target type list.
When the connection mode to Hive is Embedded, the Job is run in your local machine and calls this jar installed in the Studio.
When the connection mode to Hive is Standalone, the Job is run in the server hosting Hive and this jar file is sent to the HDFS system of the cluster you are connecting to. Therefore, ensure that you have properly defined the NameNode URI in the corresponding field of the Basic settings view.
If you are using the LOAD action, tHiveLoad copies the local data to the target table.
If you are using the INSERT action, tHiveLoad copies data to a local directory.
If you leave this Local check box clear, the directory defined in the File path field is assumed to be in the HDFS system to be used and data will be moved to the target location.
Auto install: at runtime, the Job uploads and deploys the Tez libraries provided by the Studio into the directory you specified in the Install folder in HDFS field, for example, /tmp/usr/tez.
If you have set the tez.lib.uris property in the properties table, this directory overrides the value of that property at runtime. But the other properties set in the properties table are still effective.
Use exist: the Job accesses the Tez libraries already deployed in the Hadoop cluster to be used. You need to enter the path pointing to those libraries in the Lib path (folder or file) field.
Lib jar: this table appears when you have selected Auto install from the Tez lib list and the distribution you are using is Custom. In this table, you need to add the Tez libraries to be uploaded.
Note that if you are using the centrally stored metadata from the Repository, this table automatically inherits the properties defined in that metadata and becomes uneditable unless you change the Property type from Repository to Built-in.
Typically, the HDFS-related properties can be found in the hdfs-default.xml file of your distribution, such as http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.
Apache also provides a page to list the Hive-related properties: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties.
Note that if you are using the centrally stored metadata from the Repository, this table automatically inherits the properties defined in that metadata and becomes uneditable unless you change the Property type from Repository to Built-in.
Ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to MapR's documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\ hadoop\hadoop-VERSION\lib\native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.
Without adding the specified library or libraries, you may encounter the following error: no MapRClient in java.library.path.
Set the -Djava.library.path argument, for example, in the Job Run VM arguments area of the Run/Debug view in the Preferences dialog box in the Window menu. This argument provides to the Studio the path to the native library of that MapR client. This allows the subscription-based users to make full use of the Data viewer to view locally in the Studio the data stored in MapR.