tSqoopImport Standard properties

Mode

Select the mode in which Sqoop is called in a Job execution.

Use Commandline: the Sqoop shell is used to call Sqoop. You can read data from either HDFS or HCatalog. In this mode, you have to deploy and run the Job in the host where Sqoop is installed. Therefore, if you are a subscription-based user, we recommend installing and using a Jobserver provided by Talend in that host to run the Job; if you are using one of the Talend solutions with Big Data, you have to ensure that the Studio and the Sqoop to be used are in the same machine.

Use Java API: the Java API is used to call Sqoop. In this mode, the Job can be run locally in the Studio but you need to configure the connection to the Hadoop distribution to be used. Note that JDK is required to execute the Job in the Java API mode and the versions of the JDK kits installed in both machines must be compatible with each other; for example, the versions are the same or the JDK version of the Hadoop machine is more recent.

Hadoop properties

Distribution

1. Select Import from existing version to import an officially supported distribution as base and then add other required jar files which the base distribution does not provide.

2. Select Import from zip to import the configuration zip for the custom distribution to be used. This zip file should contain the libraries of the different Hadoop elements and the index file of these libraries.

   Note that custom versions are not officially supported by Talend . Talend and its community provide you with the opportunity to connect to custom versions from the Studio but cannot guarantee that the configuration of whichever version you choose will be easy, due to the wide range of different Hadoop distributions and versions that are available. As such, you should only attempt to set up such a connection if you have sufficient Hadoop experience to handle any issues on your own.

   Information noteNote:

   In this dialog box, the active check box must be kept selected so as to import the jar files pertinent to the connection to be created between the custom distribution and this component.

   For a step-by-step example about how to connect to a custom distribution and share this connection, see Hortonworks.

Hadoop Version

Select the version of the Hadoop distribution you are using. The available options vary depending on the component you are using.

NameNode URI

Type in the URI of the Hadoop NameNode, the master node of a Hadoop system. For example, we assume that you have chosen a machine called masternode as the NameNode, then the location is hdfs://masternode:portnumber. If you are using WebHDFS, the location should be webhdfs://masternode:portnumber; WebHDFS with SSL is not supported yet.

JobTracker Host

Select this check box and in the displayed field, enter the location of the ResourceManager of your distribution. For example, tal-qa114.talend.lan:8050.

This property is required when the query you want to use is executed in Windows and it is a Select query. For example, SELECT your_column_name FROM your_table_name

For further information about the Hadoop Map/Reduce framework, see the Map/Reduce tutorial in Apache's Hadoop documentation on http://hadoop.apache.org.

Use kerberos authentication

In addition, since this component performs Map/Reduce computations, you also need to authenticate the related services such as the Job history server and the Resource manager or Jobtracker depending on your distribution in the corresponding field. These principals can be found in the configuration files of your distribution. For example, in a CDH4 distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

This check box is available depending on the Hadoop distribution you are connecting to.

Select the Use a keytab to authenticate check box to log into a Kerberos-enabled system using a given keytab file. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend Jobserver.

Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the username you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

Hadoop user name

Enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed. Note that this field is available depending on the distribution you are using.

JDBC property

Connection

Enter the JDBC URL used to connect to the database where the source data is stored.

User name and Password

Enter the authentication information used to connect to the source database.

To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

Driver JAR

In either the Use Commandline mode or the Java API mode, you must add the driver file of the database to be used to the lib folder of the Hadoop distribution you are using. For that purpose, use this Driver JAR table to add that driver file for the current Job you are designing.

Class name

Enter the class name for the specified driver between double quotation marks. For example, for the RedshiftJDBC41-1.1.13.1013.jar driver, the name to be entered is com.amazon.redshift.jdbc41.Driver.

ORA-00933: SQL command not properly ended

Table Name

Type in the name of the table to be transferred into HDFS.

This field is not available when you are using the free-form query mode by selecting the Use query check box.

File format

Delete target directory

Select this check box to remove the target directory of the transfer.

Append

Select this check box to append transferred data to an existing dataset in HDFS.

Compress

Select this check box to enable compression.

Direct

Select this check box to use the import fast path.

Specify columns

Select this check box to display the column table where you can specify the columns you want to transfer into HDFS.

Use WHERE clause

Select this check box to use a WHERE clause that controls the rows to be transferred. In the field displayed, you can type in the condition used to select the rows you want. For example, type in id >400 to import only the rows where the id column has a value greater than 400.

Use query

Select this check box to use the free-form query mode provided by Sqoop.

Once selecting it, you are able to enter the free-form query you need to use.

Then, you must specify the target directory and if the Sqoop imports data in parallel, specify as well the Split by argument.

Specify Target Dir

Select this check box to enter the path to the target location, in HDFS, where you want to transfer the source data to.

This location should be a new directory; otherwise, you must select the Append check box.

Specify Split by

Select this check box, then, enter the table column you need and are able to use as the splitting column to split the workload.

For example, for a table where the id column is the key column, enter tablename.id. Then Sqoop will split the data to be transferred according to their ID values and imports them in parallel.

Specify Number of Mappers

Select this check box to indicate the number of map tasks (parallel processes) used to perform the data transfer.

If you do not want Sqoop to work in parallel, enter 1 in the displayed field.

Print Log

Select this check box to activate the Verbose check box.

Verbose

Select this check box to print more information while working, for example, the debugging information.

Use MySQL default delimiters

Select this check box to use MySQL's default delimiter set. This check box is available only to the Commandline mode.

Define Java mapping

Sqoop provides default configuration that maps most SQL types to appropriate Java types. If you need to use your custom map to overwrite the default ones at runtime, select this check box and define the map(s) you want to use in the table that appears.

Define Hive mapping

Sqoop provides default configuration that maps most SQL types to appropriate Hive types. If you need to use your custom map to overwrite the default ones at runtime, select this check box and define the map(s) you want to use in the table that appears.

Additional arguments

Complete this table to use additional arguments if needs be.

By adding additional arguments, you are able to perform multiple operations in one single transaction. For example, you can use --hive-import and --hive-table in the Commandline mode or hive.import and hive.table.name in the Java API mode to create Hive table and write data in at the runtime of the transaction writing data in HDFS. For further information about the available Sqoop arguments in the Commandline mode and the Java API mode, respectively, see Additional arguments.

For more information about generic and specific arguments, see Using Generic and Specific Arguments in the offical Sqoop documentation.

Use speed parallel data transfers

Available in the Use Commandline mode only.

Hadoop properties

Mapred job map memory mb and Mapred job reduce memory mb

You can tune the map and reduce computations by selecting the Set memory check box to set proper memory allocations for the computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred job map memory mb and the Mapred job reduce memory mb fields, respectively. By default, the values are both 1024 which are normally appropriate for running the computations.

The memory parameters to be set are Map (in Mb), Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate memory to the map and the reduce computations and the ApplicationMaster of YARN.

Path separator in server

Leave the default value of the Path separator in server as it is, unless you have changed the separator used by your Hadoop distribution's host machine for its PATH variable or in other words, that separator is not a colon (:). In that situation, you must change this value to the one you are using in that host.

tStatCatcher Statistics

Select this check box to collect log data at the component level.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

EXIT_CODE: the exit code of the remote command. This is an After variable and it returns an integer.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage rule

This component is used standalone. It respects the Sqoop prerequisites. You need necessary knowledge about Sqoop to use it.

We recommend using the Sqoop of version 1.4+ in order to benefit the full functions of these components.

For further information about Sqoop, see the Sqoop manual on: http://sqoop.apache.org/docs/

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction with Talend Studio . The following list presents MapR related information for example.

• Ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to MapR's documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\ hadoop\hadoop-VERSION\lib\native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.

  Without adding the specified library or libraries, you may encounter the following error: no MapRClient in java.library.path.

• Set the -Djava.library.path argument, for example, in the Job Run VM arguments area of the Run/Debug view in the Preferences dialog box in the Window menu. This argument provides to the Studio the path to the native library of that MapR client. This allows the subscription-based users to make full use of the Data viewer to view locally in the Studio the data stored in MapR.

For further information about how to install a Hadoop distribution, see the manuals corresponding to the Hadoop distribution you are using.

Limitation

If you have selected the Use Commandline mode, you need to use the host where Sqoop is installed to run the Job using this component.

Connections

Outgoing links (from this component to another):

Trigger: Run if; On Subjob Ok; On Subjob Error.

Incoming links (from one component to this one):

Row: Iterate;

Trigger: Run if; On Subjob Ok; On Subjob Error; On Component Ok; On Component Error

For further information regarding connections, see Talend Studio User Guide.