tHiveLoad Standard properties

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the properties are stored.

Use an existing connection

Select this check box and in the Component List drop-down list, select the desired connection component to reuse the connection details you already defined.

Distribution

1. Select Import from existing version to import an officially supported distribution as base and then add other required jar files which the base distribution does not provide.

2. Select Import from zip to import the configuration zip for the custom distribution to be used. This zip file should contain the libraries of the different Hadoop elements and the index file of these libraries.

   Note that custom versions are not officially supported by Talend . Talend and its community provide you with the opportunity to connect to custom versions from Talend Studio but cannot guarantee that the configuration of whichever version you choose will be easy, due to the wide range of different Hadoop distributions and versions that are available. As such, you should only attempt to set up such a connection if you have sufficient Hadoop experience to handle any issues on your own.

   Information noteNote:

   In this dialog box, the active check box must be kept selected so as to import the jar files pertinent to the connection to be created between the custom distribution and this component.

   For a step-by-step example about how to connect to a custom distribution and share this connection, see Hortonworks.

Hive version

Select the version of the Hadoop distribution you are using. The available options vary depending on the component you are using.

Load action

Select the action you need to carry for writing data into the specified destination.

• When you select LOAD, you are moving or copying data from a directory you specify.

• When you select INSERT, you are moving or copying data based on queries.

Execution engine

Select this check box and from the drop-down list, select the framework you need to use to perform the INSERT action.

Before using Tez, ensure that the Hadoop cluster you are using supports Tez. You will need to configure the access to the relevant Tez libraries via the Advanced settings view of this component.

For further information about Hive on Tez, see Apache's related documentation. Some examples are presented there to show how Tez can be used to gain performance over MapReduce.

Target type

This drop-down list appears only when you have selected INSERT from the Load action list.

Select from this list the type of the location you need to write data in.

• If you select Table as destination, you can still choose to append data to or overwrite the contents in the specified table.

• If you select Directory as destination, you are overwriting the contents in the specified directory

Table name

Enter the name of the Hive table you need to write data in.

Note that with the INSERT action, this field is available only when you have selected Table from the Target type list.

File path

Enter the directory you need to read data from or write data in, depending on the action you have selected from the Load action list.

• If you have selected LOAD: this is the path to the data you want to copy or move into the specified Hive table.

• If you have selected INSERT: this is the directory to which you want to export data from a Hive table. With this action, the File path field is available only when you have selected Directory from the Target type list.

The target table uses the Parquet format

If the table in which you need to write data is a PARQUET table, select this check box.

Then from the Compression list that appears, select the compression mode you need to use to handle the PARQUET file. The default mode is Uncompressed.

Action on file

Select the action to be carried out for writing data.

This list is available only when the target is a Hive table; if the target is a directory, the action to be used is automatically OVERWRITE.

Query

This field appears when you have selected INSERT from the Load action list.

Enter the appropriate query for selecting the data to be exported to the specified Hive table or directory.

Local

Select this check box to use the Hive LOCAL statement for accessing a local directory. Note that this local directory is actually in the machine in which the Job is run. Therefore, when the connection mode to Hive is Standalone, the Job is run in the machine where the Hive application is installed and thus this local directory is in that machine.

This statement is used along with the directory you have defined in the File path field. Therefore, this Local check box is available only when the File path field is available.

• If you are using the LOAD action, tHiveLoad copies the local data to the target table.

• If you are using the INSERT action, tHiveLoad copies data to a local directory.

• If you leave this Local check box clear, the directory defined in the File path field is assumed to be in the HDFS system to be used and data will be moved to the target location.

For further information about this LOCAL statement, see Apache's documentation about Hive's Language.

Select this check box to use the Hive Partition clause in loading or inserting data in a Hive table. You need to enter the partition keys and their values to be used in the field that appears.

For example, enter contry='US', state='CA'. This makes a partition clause reading Partition (contry='US', state='CA'), that is to say, a US and CA partition.

Also, it is recommended to select the Create partition if not exist check box that appears to ensure that you will not create a duplicate partition.

Die on error

Select this check box to kill the Job when an error occurs.