# Defining the MapR connection parameters

Complete the MapR connection configuration in the Spark configuration tab of the Run view of your Job. This configuration is effective on a per-Job basis.

The information in this section is only for users who have subscribed to Talend Data Fabric or to any Talend product with Big Data.

## Procedure

1. Enter the basic configuration information:
   | Use local<br>timezone | Select this check box to let Spark use the local time zone provided by<br>the system.<br><br>Information noteNote:<br><br><ul> <li>If you clear this check box, Spark use UTC time zone.</li> <li>Some components also have the <span>Use local timezone for                                    date</span> check box. If you clear the check box from the                                 component, it inherits time zone from the Spark configuration.</li> </ul> |
   | --- | --- |
   | Use dataset API in<br>migrated components | Select this check box to let the components use Dataset (DS) API<br>instead of Resilient Distributed Dataset (RDD) API:<br><br><ul> <li>If you select the check box, the components inside the Job run with                              DS which improves performance.</li> <li>If you clear the check box, the components inside the Job run with                              RDD which means the Job remains unchanged. This ensures the backwards                              compatibility.</li> </ul><br><br>This check box is selected by default, but if you import a Job from 7.3 backwards, the<br>check box will be cleared as those Jobs run with RDD.<br><br>Information noteImportant: If your Job contains tDeltaLakeInput and<br>tDeltaLakeOutput components, you must select this check box. |
   | Use timestamp for<br>dataset components | Select this check box to use java.sql.Timestamp for dates.<br><br>Information noteNote: If you leave this check box clear, java.sql.Timestamp or java.sql.Date can be used<br>depending on the pattern. |
2. Select the type of the Spark cluster you need to connect to.
   | Standalone | Talend Studio connects to a Spark-enabled cluster to run the Job from this cluster.<br><br>If you are using the Standalone mode, you need to<br>set the following parameters:<br><br><ul> <li> <p>In the <span>Spark host</span> field, enter the URI                of the Spark Master of the Hadoop cluster to be used.</p> </li> <li> <p>In the <span>Spark home</span> field, enter the                location of the Spark executable installed in the Hadoop cluster to be used.</p> </li> <li> <p> <span>If the Spark cluster cannot recognize the machine in which the Job is                    launched, select this <span>Define the driver hostname or IP                        address</span> check box and enter the host name or the IP address of                    this machine. This allows the Spark master and its workers to recognize this                    machine to find the Job and thus its driver.</span> </p> <p>Note that in this situation, you also need to add the name and the IP                address of this machine to its <em>host</em> file.</p> </li> </ul> |
   | --- | --- |
   | Yarn client | Talend Studio runs the Spark driver to orchestrate how the Job should be performed and<br>then send the orchestration to the Yarn service of a given Hadoop cluster so<br>that the Resource Manager of this Yarn service requests execution resources<br>accordingly.<br><br>If you are using the Yarn client<br>mode, you need to set the following parameters in their corresponding<br>fields (if you leave the check box of a service clear, then at runtime,<br>the configuration about this parameter in the Hadoop cluster to be used<br>will be ignored):<br><br><ul> <li> <p>In the <span>Resource manager</span>Use datanode                    field, enter the address of the ResourceManager service of the Hadoop cluster to                    be used.</p> </li> <li> <p>Select the <span>Set resourcemanager                        scheduler address</span> check box and enter the Scheduler address in                    the field that appears.</p> </li> <li> <p>Select the <span>Set jobhistory                        address</span> check box and enter the location of the JobHistory                    server of the Hadoop cluster to be used. This allows the metrics information of                    the current Job to be stored in that JobHistory server.</p> </li> <li> <p>Select the <span>Set staging                        directory</span> check box and enter this directory defined in your                    Hadoop cluster for temporary files created by running programs. Typically, this                    directory can be found under the <em>yarn.app.mapreduce.am.staging-dir</em> property in the configuration files                    such as <em>yarn-site.xml</em> or <em>mapred-site.xml</em> of your distribution.</p> </li> <li> <p>If you are accessing the Hadoop cluster running with Kerberos security,                select this checkbox, then, enter the Kerberos principal names for the Resource                Manager service and the Job History service in the displayed fields. This enables                you to use your username to authenticate against the credentials stored in Kerberos.                These principals can be found in the configuration files of your distribution, such                as in <em>yarn-site.xml</em> and in <em>mapred-site.xml</em>. </p> <p>If you need to use a Kerberos keytab file to log in, select <span>Use a keytab to authenticate</span>. A keytab file contains                pairs of Kerberos principals and encrypted keys. You need to enter the principal to                be used in the <span>Principal</span> field and the access                path to the keytab file itself in the <span>Keytab</span>                field. This keytab file must be stored in the machine in which your Job actually                runs, for example, on a <span>Talend JobServer</span>.</p> <p>Note that the user that executes a keytab-enabled Job is not necessarily                the one a principal designates but must have the right to read the keytab file being                used. For example, the username you are using to execute a Job is <em>user1</em> and the principal to be used is <em>guest</em>; in this                situation, ensure that <em>user1</em> has the right to read the keytab                file to be used.</p> </li> <li> <p>The <span>User name</span> field is available when you            are not using Kerberos to authenticate. In the <span>User               name</span> field, enter the login username for your distribution. If you leave            it empty, the username of the machine hosting <span>Talend Studio</span> will            be used.</p> </li> <li> <p> <span>If the Spark cluster cannot recognize the machine in which the Job is                    launched, select this <span>Define the driver hostname or IP                        address</span> check box and enter the host name or the IP address of                    this machine. This allows the Spark master and its workers to recognize this                    machine to find the Job and thus its driver.</span> </p> <p>Note that in this situation, you also need to add the name and the IP                address of this machine to its <em>host</em> file.</p> </li> </ul> |
   | Yarn cluster | The Spark driver runs in your Yarn cluster to orchestrate how the Job<br>should be performed.<br><br>If you are using the Yarn cluster mode, you need<br>to define the following parameters in their corresponding fields (if you<br>leave the check box of a service clear, then at runtime, the<br>configuration about this parameter in the Hadoop cluster to be used will<br>be ignored):<br><br><ul> <li> <p>In the <span>Resource manager</span>Use datanode                    field, enter the address of the ResourceManager service of the Hadoop cluster to                    be used.</p> </li> <li> <p>Select the <span>Set resourcemanager                        scheduler address</span> check box and enter the Scheduler address in                    the field that appears.</p> </li> <li> <p>Select the <span>Set jobhistory                        address</span> check box and enter the location of the JobHistory                    server of the Hadoop cluster to be used. This allows the metrics information of                    the current Job to be stored in that JobHistory server.</p> </li> <li> <p>Select the <span>Set staging                        directory</span> check box and enter this directory defined in your                    Hadoop cluster for temporary files created by running programs. Typically, this                    directory can be found under the <em>yarn.app.mapreduce.am.staging-dir</em> property in the configuration files                    such as <em>yarn-site.xml</em> or <em>mapred-site.xml</em> of your distribution.</p> </li> <li> <p> <span> <span>Set path to custom Hadoop                                            configuration JAR</span>: if you are using                                        connections defined in <span>Repository</span> to                                        connect to your Cloudera or Hortonworks cluster, you can                                        select this check box in the                                            <span>Repository</span> wizard and in the                                        field that is displayed, specify the path to the JAR file                                        that provides the connection parameters of your Hadoop                                        environment. Note that this file must be accessible from the                                        machine where you Job is launched.</span> </p> <div>This kind of Hadoop configuration JAR file is                           automatically generated when you build a big data Job from <span>Talend Studio</span>. This JAR file is by default named with this                           pattern:<pre><code><pre><code><code>hadoop-conf-[name_of_the_metadata_in_the_repository]_[name_of_the_context].jar</code></code></pre></code></pre>You                           can also download this JAR file from the web console of your cluster or                           simply create a JAR file yourself by putting the configuration files in                           the root of your JAR file. For example:<pre><code><pre><code><code>hdfs-sidt.xmlcore-site.xml</code></code></pre></code></pre> </div> <p>The parameters from your custom JAR file override the parameters                                    you put in the <span>Spark configuration</span> field.                                    They also override the configuration you set in the                                    configuration components such as                                        <span>tHDFSConfiguration</span> or                                        <span>tHBaseConfiguration</span> when the related                                    storage system such as HDFS, HBase or Hive are native to Hadoop.                                    But they do not override the configuration set in the                                    configuration components for the third-party storage system such                                    as tAzureFSConfiguration.</p> </li> <li> <p>If you are accessing the Hadoop cluster running with Kerberos security,                select this checkbox, then, enter the Kerberos principal names for the Resource                Manager service and the Job History service in the displayed fields. This enables                you to use your username to authenticate against the credentials stored in Kerberos.                These principals can be found in the configuration files of your distribution, such                as in <em>yarn-site.xml</em> and in <em>mapred-site.xml</em>. </p> <p>If you need to use a Kerberos keytab file to log in, select <span>Use a keytab to authenticate</span>. A keytab file contains                pairs of Kerberos principals and encrypted keys. You need to enter the principal to                be used in the <span>Principal</span> field and the access                path to the keytab file itself in the <span>Keytab</span>                field. This keytab file must be stored in the machine in which your Job actually                runs, for example, on a <span>Talend JobServer</span>.</p> <p>Note that the user that executes a keytab-enabled Job is not necessarily                the one a principal designates but must have the right to read the keytab file being                used. For example, the username you are using to execute a Job is <em>user1</em> and the principal to be used is <em>guest</em>; in this                situation, ensure that <em>user1</em> has the right to read the keytab                file to be used.</p> </li> <li> <p>The <span>User name</span> field is available when you            are not using Kerberos to authenticate. In the <span>User               name</span> field, enter the login username for your distribution. If you leave            it empty, the username of the machine hosting <span>Talend Studio</span> will            be used.</p> </li> <li> <p> <span>Select the <span>Wait for the Job to                  complete</span> check box to make <span>Talend Studio</span> or,               if you use <span>Talend JobServer</span>,               your Job JVM keep monitoring the Job until the execution of the Job is over. By               selecting this check box, you actually set the                  <span>spark.yarn.submit.waitAppCompletion</span> property to be true.               While it is generally useful to select this check box when running a Spark Batch Job,               it makes more sense to keep this check box clear when running a Spark Streaming               Job.</span> </p> </li> </ul> |

   Ensure that the username in the Yarn client mode is the same one you put in tHDFSConfiguration, the component used to provide HDFS connection information to Spark.
3. With the Yarn client mode, the Property type list is displayed to allow you to select an established Hadoop connection from the Repository, on the condition that you have created this connection in the Repository. Then Talend Studio will reuse that set of connection information for this Job.
4. If you need to launch from Windows, it is recommended to specify where the *winutils.exe* program to be used is stored.
   * If you know where to find your *winutils.exe* file and you want to use it, select the Define the Hadoop home directory check box and enter the directory where your *winutils.exe* is stored.

   * Otherwise, leave this check box clear, Talend Studio generates one by itself and automatically uses it for this Job.
5. Verify, for example with your cluster administrator, whether your MapR cluster is secured with the MapR ticket authentication mechanism.
   * If the MapR cluster to be used is secured with the MapR ticket authentication mechanism, set the MapR ticket authentication configuration by following the explanation in [Setting up the MapR ticket authentication](https://help.qlik.com/talend/en-US/components/8.0/mapr/setting-up-mapr-ticket-authentication-standard-component-the-mapr).

   * Otherwise, leave the Use MapR Ticket authentication check box clear.
6. In the Spark "scratch" directory field, enter the directory in which Talend Studio stores in the local system the temporary files such as the jar files to be transferred. If you launch the Job on Windows, the default disk is *C:*. As a result, if you leave */tmp* in this field, this directory is *C:/tmp*.

## Results

* After the connection is configured, you can tune the Spark performance, although not required, by following the process explained in:
  * [Tuning Spark for Apache Spark Batch Jobs](https://help.qlik.com/talend/r/en-US/8.0/components/spark-batch/tuning-spark-for-apache-spark-batch-jobs-spark-batch-component) for Spark Batch Jobs.

  * [Tuning Spark for Apache Spark Streaming Jobs](https://help.qlik.com/talend/r/en-US/8.0/components/spark-streaming/tuning-spark-for-apache-spark-streaming-jobs-spark-stream-component) for Spark Streaming Jobs.
* It is recommended to activate the Spark logging and checkpointing system in the Spark configuration tab of the Run view of your Spark Job, in order to help debug and resume your Spark Job when issues arise:
  * [Logging and checkpointing the activities of your Apache Spark Job](https://help.qlik.com/talend/r/en-US/8.0/components/spark-batch/logging-and-checkpointing-activities-of-your-apache-spark-job-spark-batch-component-if-you).