# tHDFSConnection

Connects to a given HDFS so that the other Hadoop components can reuse the connection it creates to communicate with this HDFS.

tHDFSConnection provides connection to the Hadoop distributed file system (HDFS) of interest at runtime.

This component is not shipped with your Talend Studio by default. You need to install it using the Feature Manager. For more information, see [Installing features using the Feature Manager](https://help.qlik.com/talend/r/en-US/8.0/studio-user-guide/install-features-to-talend-studio).

## tHDFSConnection Standard properties

These properties are used to configure tHDFSConnection running in the Standard Job framework.

The Standard tHDFSConnection component belongs to the Big Data and the File families.

The component in this framework is available in all Talend products with Big Data and in Talend Data Fabric.

## Basic settings

| Properties | Description |
| --- | --- |
| Property Type | Select the way the connection details<br>will be set.<br><br><ul> <li> <p> <span>Built-In</span>: The connection details will be set                                locally for this component. You need to specify the values for all                                related connection properties manually.</p> </li> <li> <p> <span>Repository</span>: The connection details stored                                centrally in <span><span>Repository</span><abbr> &gt; </abbr><span>Metadata</span></span> will be reused by this component. </p> <p>You need to click                                the <span>[...]</span> button next to it and in the pop-up                                    <span>Repository Content</span> dialog box, select the                                connection details to be reused, and all related connection                                properties will be automatically filled in.</p> </li> </ul> |
| Hadoop version | Select the Hadoop version of the distribution you are using. |
| Scheme | Select the URI scheme of the file system to be used from the<br>Scheme drop-down list. This scheme could be<br><br><ul> <li>HDFS</li> <li>WebHDFS. WebHDFS with SSL is not supported yet.</li> <li>ADLS. Only Azure Data Lake Storage Gen1 is supported.</li> </ul><br><br>The schemes present on this list vary depending on the distribution you<br>are using and only the scheme that appears on this list with a given<br>distribution is officially supported by Talend.<br><br>Once a scheme is<br>selected, the corresponding syntax such as<br>webhdfs://localhost:50070/ for WebHDFS is displayed in the<br>field for the NameNode location of your cluster.<br><br>If you have selected<br>ADLS, the connection parameters to be defined become:<br><br><ul> <li> <p> <span>In the <span>Client                                    ID</span> and the <span>Client                                    key</span> fields, enter, respectively, the authentication                                ID and the authentication key generated upon the registration of the                                application that the current Job you are developing uses to access                                Azure Data Lake Storage.</span> </p> <p>Ensure that the application to be used                            has appropriate permissions to access Azure Data Lake. You can check                            this on the Required permissions view of this application on Azure. For                            further information, see Azure documentation <a href="https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-service-to-service-authenticate-using-active-directory#step-3-assign-the-azure-ad-application-to-the-azure-data-lake-store-account-file-or-folder" target="_blank">Assign the Azure AD application to                                the Azure Data Lake Storage account file or folder</a>.</p> </li> <li> <p> <span>In the <span>Token                                    endpoint</span> field, copy-paste the OAuth 2.0 token                                endpoint that you can obtain from the <span>Endpoints</span> list accessible on the <span>App registrations</span> page on your                                Azure portal.</span> </p> </li> </ul><br><br><br>For a<br>video demonstration, see [Configure and use Azure in a<br>Job](https://www.youtube.com/watch?v=Ww9-sp5Kd1o). |
| Use Knox | Select this check box to enable support for Apache Knox when the<br>Scheme is set to WebHDFS. When<br>selected, additional Knox-specific options become available for secure gateway access.<br><br><ul> <li><span>Knox Gateway</span>: Enter the URL of the Knox gateway.</li> <li><span>User</span>: Enter the Knox username for authentication.</li> <li><span>Password</span>: Enter the Knox password for authentication.</li> <li><span>Set Knox timeout (s)</span>: Option to define a custom connection                  timeout. When enabled, the <span>Knox timeout</span> field appears.</li> </ul><br><br><br>The following options are hidden when Use Knox is<br>selected:<br><br><ul> <li><span>NameNode URI</span></li> <li><span>Inspect the classpath for configurations</span></li> <li><span>Use kerberos authentication</span></li> <li><span>Use MapR Ticket authentication</span></li> <li><span>User name</span></li> <li><span>Hadoop properties</span></li> <li><span>Use Datanode Hostname</span></li> <li><span>Setup HDFS Encryption Configurations</span></li> </ul> |
| NameNode URI | Type in the URI of the Hadoop NameNode, the master node of a<br>Hadoop system. For example, we assume that you have chosen a machine called *masternode* as the NameNode, then the location is<br>hdfs://masternode:portnumber. If you are using WebHDFS, the location should be<br>webhdfs://masternode:portnumber; WebHDFS with SSL is not<br>supported yet. |
| Inspect the classpath for configurations | Select this check box to allow the component to check the configuration<br>files in the directory you have set with the *$HADOOP\_CONF\_DIR*<br>variable and directly read parameters from these files in this directory. This<br>feature allows you to easily change the Hadoop configuration for the component to<br>switch between different environments, for example, from a test environment to a<br>production environment.<br><br>In this situation, the fields or options used to configure Hadoop<br>connection and/or Kerberos security are hidden.<br><br>If you want to use certain parameters such as the Kerberos parameters but<br>these parameters are not included in these Hadoop configuration files, you need to<br>create a file called *talend-site.xml* and put this file into the<br>same directory defined with *$HADOOP\_CONF\_DIR*. This *talend-site.xml* file should read as follows:<br><br>```<br><!-- Put site-specific property overrides in this file. --><br><configuration><br><property><br><name>talend.kerberos.authentication </name><br><value>kinit </value><br><description> Set the Kerberos authentication method to use. Valid values are: kinit or keytab.  </description><br></property><br><property><br><name>talend.kerberos.keytab.principal </name><br><value>user@BIGDATA.COM </value><br><description> Set the keytab's principal name.  </description><br></property><br><property><br><name>talend.kerberos.keytab.path </name><br><value>/kdc/user.keytab </value><br><description> Set the keytab's path.  </description><br></property><br><property><br><name>talend.encryption </name><br><value>none </value><br><description> Set the encryption method to use. Valid values are: none or ssl.  </description><br></property><br><property><br><name>talend.ssl.trustStore.path </name><br><value>ssl </value><br><description> Set SSL trust store path.  </description><br></property><br><property><br><name>talend.ssl.trustStore.password </name><br><value>ssl </value><br><description> Set SSL trust store password.  </description><br></property><br></configuration><br>```<br><br>The parameters read from these configuration files override the default<br>ones used by Talend Studio.<br>When a parameter does not exist in these configuration files, the default one is<br>used. |
| Use kerberos authentication | If you are accessing the Hadoop cluster running with<br>Kerberos security, select this checkbox, then, enter the Kerberos principal name<br>for the NameNode in the field displayed. This enables you to use your username<br>to authenticate against the credentials stored in Kerberos.<br><br>This check box is available depending on the Hadoop distribution you are<br>connecting to. |
| Use MapR Ticket authentication | Select this checkbox to authenticate using MapR Ticket, and enter the<br>following information:<br><br><ul> <li><span>User name</span>: Enter your username.</li> <li><span>Group</span>: Enter the name of the group.</li> <li><span>Password</span>: Enter your password to authenticate to                                MapR Ticket.</li> <li><span>Cluster name</span>: Enter the name of the cluster you want                                to use.</li> <li><span>Ticket duration</span>: Enter the duration of the ticket in seconds. For example:                                <span>86400L</span>, where <span>L</span> corresponds to a                                long integer.</li> </ul> |
| Use a keytab to authenticate | Select the Use a keytab to authenticate<br>checkbox to log into a Kerberos-enabled system using a given keytab file. A keytab<br>file contains pairs of Kerberos principals and encrypted keys. You need to enter the<br>principal to be used in the Principal field and<br>the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in<br>which your Job actually runs, for example, on a Talend JobServer.<br><br>Note that the user that executes a keytab-enabled Job is not necessarily<br>the one a principal designates but must have the right to read the keytab file being<br>used. For example, the username you are using to execute a Job is *user1* and the principal to be used is *guest*; in this<br>situation, ensure that *user1* has the right to read the keytab<br>file to be used. |
| User name | User authentication name of HDFS. |
| Group | Enter the membership including the authentication user under which the HDFS<br>instances were started. This field is available depending on the distribution you<br>are using. |
| Hadoop properties | *Talend Studio* uses a default configuration for its engine to perform<br>operations in a Hadoop distribution. If you need to use a custom configuration in a specific<br>situation, complete this table with the property or properties to be customized. Then at<br>runtime, the customized property or properties will override those default ones.<br><br><ul> <li> <p>Note that if you are using the centrally stored metadata from the <span>Repository</span>, this table automatically inherits the					properties defined in that metadata and becomes uneditable unless you change the						<span>Property type</span> from <span>Repository</span> to <span>Built-in</span>.</p> </li> </ul><br><br>For further information about the properties required by Hadoop and its related<br>systems such as HDFS and Hive, see the documentation of the Hadoop distribution you are<br>using or see [Apache's Hadoop documentation](http://hadoop.apache.org/docs) and then select the version of the<br>documentation you want. For demonstration purposes, the links to some properties are listed<br>below:<br><br><ul> <li> <p>Typically, the HDFS-related properties can be found in the <em>hdfs-default.xml</em> file of your distribution, such as <a href="http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml" target="_blank">                     http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml</a>.</p> </li> <li> <p>Apache also provides a page to list the <a href="https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties" target="_blank">Hive-related properties</a>.</p> </li> </ul> |
| Use datanode hostname | Select the Use datanode hostname check box<br>to allow the Job to access datanodes via their hostnames. This actually sets the *dfs.client.use.datanode.hostname* property to *true*. |
| Setup HDFS encryption configurations | If the HDFS transparent encryption has been enabled in your cluster, select<br>the Setup HDFS encryption configurations check<br>box and in the HDFS encryption key provider field<br>that is displayed, enter the location of the KMS proxy.<br><br>For further information about the HDFS transparent encryption and its KMS proxy, see [Transparent Encryption in HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html). |

## Advanced settings

| Properties | Description |
| --- | --- |
| tStatCatcher Statistics | Select this check box to collect log data at the component level. |

## Global Variables

| Variables | Description |
| --- | --- |
| Global Variables | ERROR\_MESSAGE: the error message generated by the<br>component when an error occurs. This is an After variable and it returns a string. This<br>variable functions only if the Die on error checkbox is<br>cleared, if the component has this checkbox.<br><br>A Flow variable functions during the execution of a component<br>while an After variable functions after the execution of the component.<br><br>To fill up a field or expression with a variable, press Ctrl+Space to access the variable list and choose the variable to use from<br>it.<br><br>For more information about variables, see [Using contexts and variables](https://help.qlik.com/talend/r/en-US/8.0/studio-user-guide/contexts-and-variables). |

## Usage

| Usage guidance | Description |
| --- | --- |
| **Usage rule** | This component is generally used with other Hadoop components. |
| Prerequisites | The Hadoop distribution must be properly installed, so as to guarantee the interaction<br>with *Talend Studio*. The following list presents MapR related information for<br>example.<br><br><ul> <li> <p>Ensure that you have installed the MapR client in the machine where                  <span>Talend Studio</span> is,               and added the MapR client library to the PATH variable of that machine. According to               MapR's documentation, the library or libraries of a MapR client corresponding to each               OS version can be found under <em>MAPR_INSTALL\                  hadoop\hadoop-VERSION\lib\native</em>. For example, the library for Windows is <em>\lib\native\MapRClient.dll</em> in the MapR client jar file.</p> <p>Without adding the specified library or libraries, you may encounter the following				error: <span>no MapRClient in java.library.path</span>.</p> </li> <li> <p>Set the <span>-Djava.library.path</span> argument,               for example, in the <span>Job Run VM arguments</span> area               of the <span>Run/Debug</span> view in the <span>Preferences</span> dialog box in the <span>Window</span> menu. This argument provides to <span>Talend Studio</span> the               path to the native library of that MapR client. This allows the subscription-based               users to make full use of the <span>Data viewer</span> to               view locally in <span>Talend Studio</span> the               data stored in MapR.</p> </li> </ul><br><br>For further information about how to install a Hadoop distribution, see the manuals<br>corresponding to the Hadoop distribution you are using. |
| Limitations | JRE 1.6+ is required. |

**Related concepts**

* [Checking the existence of a file in HDFS](https://help.qlik.com/talend/en-US/components/8.0/hdfs/thdfsexist-thdfsconnection-tmsgbox-thdfsconnection-tmsgbox-checking-existence-of-file-in-hdfs-standard-component-bigdata-in-this)
* [Iterating on a HDFS directory](https://help.qlik.com/talend/en-US/components/8.0/hdfs/thdfslist-thdfsget-thdfsconnection-iterating-on-hdfs-directory-standard-component-bigdata-this)