tPigLoad Standard properties

Property type

Either Built-In or Repository.

Built-In: No property data stored centrally.

Repository: Select the repository file where the properties are stored.

The properties are stored centrally under the Hadoop Cluster node of the Repository tree.

The fields that come after are pre-filled in using the fetched data.

For further information about the Hadoop Cluster node, see the Getting Started Guide.

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

• View schema: choose this option to view the schema only.

• Change to built-in property: choose this option to change the schema to Built-in for local changes.

• Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Built-In: You create and store the schema locally for this component only.

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Local

Click this radio button to run Pig scripts in Local mode. In this mode, all files are installed and run from your local host and file system.

Execution engine

Select the framework you need to use to perform your Pig Job.

Before using Tez, ensure that the Hadoop cluster you are using supports Tez. You will need to configure the access to the relevant Tez libraries via the Advanced settings view of this component.

For further information about Pig on Tez, see Apache's related documentation in https://cwiki.apache.org/confluence/display/PIG/Pig+on+Tez.

Distribution and Version

1. Select Import from existing version to import an officially supported distribution as base and then add other required jar files which the base distribution does not provide.

2. Select Import from zip to import the configuration zip for the custom distribution to be used. This zip file should contain the libraries of the different Hadoop elements and the index file of these libraries.

   In Talend Exchange, members of Talend community have shared some ready-for-use configuration zip files which you can download from this Hadoop configuration list and directly use them in your connection accordingly. However, because of the ongoing evolution of the different Hadoop-related projects, you might not be able to find the configuration zip corresponding to your distribution from this list; then it is recommended to use the Import from existing version option to take an existing distribution as base to add the jars required by your distribution.

   Note that custom versions are not officially supported by Talend . Talend and its community provide you with the opportunity to connect to custom versions from the Studio but cannot guarantee that the configuration of whichever version you choose will be easy, due to the wide range of different Hadoop distributions and versions that are available. As such, you should only attempt to set up such a connection if you have sufficient Hadoop experience to handle any issues on your own.

   Information noteNote:

   In this dialog box, the active check box must be kept selected so as to import the jar files pertinent to the connection to be created between the custom distribution and this component.

   For a step-by-step example about how to connect to a custom distribution and share this connection, see .

The other parameters related to the connection to your cluster could be:

• Use Kerberos authentication:

  If you are accessing the Hadoop cluster running with Kerberos security, select this check box, then, enter the Kerberos principal name for the NameNode in the field displayed. This enables you to use your user name to authenticate against the credentials stored in Kerberos.

  • If this cluster is a MapR cluster of the version 5.0.0 or later, you can set the MapR ticket authentication configuration in addition or as an alternative by following the explanation in Connecting to a security-enabled MapR.

    Keep in mind that this configuration generates a new MapR security ticket for the username defined in the Job in each execution. If you need to reuse an existing ticket issued for the same username, leave both the Force MapR ticket authentication check box and the Use Kerberos authentication check box clear, and then MapR should be able to automatically find that ticket on the fly.

  In addition, since this component performs Map/Reduce computations, you also need to authenticate the related services such as the Job history server and the Resource manager or Jobtracker depending on your distribution in the corresponding field. These principals can be found in the configuration files of your distribution. For example, in a CDH4 distribution, the Resource manager principal is set in the yarn-site.xml file and the Job history principal in the mapred-site.xml file.

  This check box is available depending on the Hadoop distribution you are connecting to.

  The HBase related principals are required by the HBaseStorage function only.

• Use a keytab to authenticate:

  Select the Use a keytab to authenticate check box to log into a Kerberos-enabled system using a given keytab file. A keytab file contains pairs of Kerberos principals and encrypted keys. You need to enter the principal to be used in the Principal field and the access path to the keytab file itself in the Keytab field. This keytab file must be stored in the machine in which your Job actually runs, for example, on a Talend Jobserver.

  Note that the user that executes a keytab-enabled Job is not necessarily the one a principal designates but must have the right to read the keytab file being used. For example, the user name you are using to execute a Job is user1 and the principal to be used is guest; in this situation, ensure that user1 has the right to read the keytab file to be used.

• NameNode URI:

  Type in the location of the NameNode corresponding to the Map/Reduce version to be used. If you are using WebHDFS, the location should be webhdfs://masternode:portnumber; WebHDFS with SSL is not supported yet.

• Resource Manager:

  Type in the location of the ResourceManager service of your cluster.

  In JobHistory, you can easily find the execution status of your Pig Job because the name of the Job is automatically created by concatenating the name of the project that contains the Job, the name and version of the Job itself and the label of the first tPigLoad component used in it. The naming convention of a Pig Job in JobHistory is ProjectName_JobNameVersion_FirstComponentName.

  Then you can continue to set the following parameters depending on the configuration of the Hadoop cluster to be used (if you leave the check box of a parameter clear, then at runtime, the configuration about this parameter in the Hadoop cluster to be used will be ignored ):

  1. Select the Set resourcemanager scheduler address check box and enter the Scheduler address in the field that appears.

  2. Select the Set jobhistory address check box and enter the location of the JobHistory server of the Hadoop cluster to be used. This allows the metrics information of the current Job to be stored in that JobHistory server.

  3. Select the Set staging directory check box and enter this directory defined in your Hadoop cluster for temporary files created by running programs. Typically, this directory can be found under the yarn.app.mapreduce.am.staging-dir property in the configuration files such as yarn-site.xml or mapred-site.xml of your distribution.

  4. Allocate proper memory volumes to the Map and the Reduce computations and the ApplicationMaster of YARN by selecting the Set memory check box in the Advanced settings view.

  5. Select the Set Hadoop user check box and enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed.

  6. Select the Use datanode hostname check box to allow the Job to access datanodes via their hostnames. This actually sets the dfs.client.use.datanode.hostname property to true. When connecting to a S3N filesystem, you must select this check box.

  For further information about these parameters, see the documentation or contact the administrator of the Hadoop cluster to be used.

• User name:

  Enter the user name under which you want to execute the Job. Since a file or a directory in Hadoop has its specific owner with appropriate read or write rights, this field allows you to execute the Job directly under the user name that has the appropriate rights to access the file or directory to be processed. Note that this field is available depending on the distribution you are using.

• Parameters related to Qubole:

  1. Click the ... button next to the API Token field to enter the authentication token generated for the Qubole user account to be used. For further information about how to obtain this token, see Manage Qubole account from the Qubole documentation.

     This token allows you to specify the user account you want to use to access Qubole. Your Job automatically uses the rights and permissions granted to this user account in Qubole.

  2. Select the Cluster label check box and enter the name of the Qubole cluster to be used. If leaving this check box clear, the default cluster is used.

     If you need details about your default cluster, ask the administrator of your Qubole service. You can also read this article from the Qubole documentaiton to find more information about configuring a default Qubole cluster.

  3. Select the Change API endpoint check box and select the region to be used. If leaving this check box clear, the default region is used.

     For further information about the Qubole Endpoints supported on QDS-on-AWS, see Supported Qubole Endpoints on Different Cloud Providers.

  4. Configure the connection to the S3 file system to be used to temporarily store the dependencies of your Job so that your Qubole cluster has access to these dependencies.

     • Access key and Secret key: enter the authentication information required to connect to the Amazon S3 bucket to be used.

       To enter the password, click the [...] button next to the password field, and then in the pop-up dialog box enter the password between double quotes and click OK to save the settings.

     • Bucket name: enter the name of the bucket in which you want to store the dependencies of your Job. This bucket must already exist on S3.
     • Temporary resource folder: enter the directory in which you want to store the dependencies of your Job. For example, enter temp_resources to write the dependencies in the /temp_resources folder in the bucket.

       If this folder already exists at runtime, its contents are overwritten by the upcoming dependencies; otherwise, this folder is automatically created.

     • Region: specify the AWS region by selecting a region name from the list. For more information about the AWS Region, see Regions and Endpoints.

WebHCat configuration

Enter the address and the authentication information of the Microsoft HD Insight cluster to be used. For example, the address could be your_hdinsight_cluster_name.azurehdinsight.net and the authentication information is your Azure account name: ychen. The Studio uses this service to submit the Job to the HD Insight cluster.

In the Job result folder field, enter the location in which you want to store the execution result of a Job in the Azure Storage to be used.

HDInsight configuration

Windows Azure Storage configuration

Enter the address and the authentication information of the Azure Storage account to be used. In this configuration, you do not define where to read or write your business data but define where to deploy your Job only. Therefore always use the Azure Storage system for this configuration.

In the Container field, enter the name of the container to be used. You can find the available containers in the Blob blade of the Azure Storage account to be used.

In the Deployment Blob field, enter the location in which you want to store the current Job and its dependent libraries in this Azure Storage account.

Inspect the classpath for configurations

Select this check box to allow the component to check the configuration files in the directory you have set with the $HADOOP_CONF_DIR variable and directly read parameters from these files in this directory. This feature allows you to easily change the Hadoop configuration for the component to switch between different environments, for example, from a test environment to a production environment.

In this situation, the fields or options used to configure Hadoop connection and/or Kerberos security are hidden.

<!-- Put site-specific property overrides in this file. --> 
<configuration> 
    <property> 
        <name>talend.kerberos.authentication </name> 
        <value>kinit </value>
         <description> Set the Kerberos authentication method to use. Valid values are: kinit or keytab.  </description> 
    </property> 
    <property> 
        <name>talend.kerberos.keytab.principal </name>
        <value>user@BIGDATA.COM </value>
        <description> Set the keytab's principal name.  </description>
    </property> 
    <property>   
        <name>talend.kerberos.keytab.path </name> 
        <value>/kdc/user.keytab </value> 
        <description> Set the keytab's path.  </description> 
    </property> 
    <property> 
        <name>talend.encryption </name> 
        <value>none </value> 
        <description> Set the encryption method to use. Valid values are: none or ssl.  </description> 
    </property> 
    <property> 
        <name>talend.ssl.trustStore.path </name> 
        <value>ssl </value> 
        <description> Set SSL trust store path.  </description> 
    </property> 
    <property> 
        <name>talend.ssl.trustStore.password </name> 
        <value>ssl </value> 
        <description> Set SSL trust store password.  </description> 
    </property> 
</configuration>

The parameters read from these configuration files override the default ones used by the Studio. When a parameter does not exist in these configuration files, the default one is used.

Load function

Input file URI

Use S3 endpoint

Select this check box to read data from a given Amazon S3 bucket folder.

HCataLog Configuration

Fill the following fields to configure HCataLog managed tables on HDFS (Hadoop distributed file system):

Distribution and Version:

HCat metastore: Enter the location of the HCatalog's metastore, which is actually Hive's metastore, a system catalog. For further information about Hive and HCatalog, see http://hive.apache.org/.

Database: The database in which tables are placed.

Table: The table in which data is stored.

Partition filter: Fill this field with the partition keys to list partitions by filter.

Enter character, string or regular expression to separate fields for the transferred data.

Compression

Select the Force to compress the output data check box to compress the data when the data is outputted by tPigStoreResult at the end of a Pig process.

Hadoop provides different compression formats that help reduce the space needed for storing files and speed up data transfer. When you need to write and compress data using the Pig program, by default you have to add a compression format as a suffix to the path pointing to the folder in which you want to write data, for example, /user/ychen/out.bz2. However, if you select this check box, the output data will be compressed even if you do not add any compression format to that path, such as /user/ychen/out.

This area is available to the HBaseStorage function. The parameters to be set are:

Zookeeper quorum:

Type in the name or the URL of the Zookeeper service you use to coordinate the transaction between your Studio and your database. Note that when you configure the Zookeeper, you might need to explicitly set the zookeeper.znode.parent property to define the path to the root znode that contains all the znodes created and used by your database; then select the Set Zookeeper znode parent check box to define this property.

Zookeeper client port:

Type in the number of the client listening port of the Zookeeper service you are using.

Table name:

Enter the name of the HBase table you need to load data from.

Load key:

Select this check box to load the row key as the first column of the result schema. In this situation, you must have created this column in the schema.

Mapping:

Complete this table to map the columns of the table to be used with the schema columns you have defined for the data flow to be processed.

Sequence Loader configuration

This area is available only to the SequenceFileLoader function. Since a SequenceFile record consists of binary key/value pairs, the parameters to be set are:

Key column:

Select the Key column of a key/value record.

Value column

Select the Value column of a key/value record.

Die on subJob error

This check box is cleared by default, meaning to skip the row on subJob error and to complete the process for error-free rows.

Tez lib

Register jar

Click the [+] button to add rows to the table and from these rows, browse to the jar files to be added. For example, in order to register a jar file called piggybank.jar, click the [+] button once to add one row, then click this row to display the [...] browse button, and click this button to browse to the piggybank.jar file following the Select Module wizard.

Use this table to define UDFs (User-Defined Functions), especially those requiring alias such as Apache DataFu Pig functions, to be executed when loading data.

Click the button to add as many rows as you need and specify an alias and a UDF in the relevant fields for each row.

If your Job includes a tPigMap component, once you have defined UDFs for this component in the tPigMap, this table is automatically filled. Likewise, once you have defined UDFs in this table, the Define functions table in the tPigMap component's Map Editor is automatically filled.

For information on how to define UDFs when mapping Pig flows, see the section on mapping Big Data flows of the Talend Open Studio for Big Data Getting Started Guide .

For more information on Apache DataFu Pig, see http://datafu.incubator.apache.org/.

Pig properties

Talend Studio uses a default configuration for its Pig engine to perform operations. If you need to use a custom configuration in a specific situation, complete this table with the property or properties to be customized. Then at runtime, the customized property or properties will override those default ones.

For example, the default_parallel key used in Pig could be set as 20.

Add and set more HBaseStorage loader options in this table. The options are:

gt: the minimum key value;

lt: the maximum key value;

gte: the minimum key value (included);

lte: the maximum key value (included);

limit: maximum number of rows to retrieve per region;

caching: number of rows to cache;

caster: the converter to use for reading values out of HBase. For example, HBaseBinaryConverter.

Define the jars to register for HCatalog

This check box appears when you are using tHCatLoader, while you can leave it clear as the Studio registers the required jar files automatically. In case any jar file is missing, you can select this check box to display the Register jar for HCatalog table and set the correct path to that missing jar.

Path separator in server

Leave the default value of the Path separator in server as it is, unless you have changed the separator used by your Hadoop distribution's host machine for its PATH variable or in other words, that separator is not a colon (:). In that situation, you must change this value to the one you are using in that host.

Mapred job map memory mb and Mapred job reduce memory mb

You can tune the map and reduce computations by selecting the Set memory check box to set proper memory allocations for the computations to be performed by the Hadoop system.

In that situation, you need to enter the values you need in the Mapred job map memory mb and the Mapred job reduce memory mb fields, respectively. By default, the values are both 1000 which are normally appropriate for running the computations.

The memory parameters to be set are Map (in Mb), Reduce (in Mb) and ApplicationMaster (in Mb). These fields allow you to dynamically allocate memory to the map and the reduce computations and the ApplicationMaster of YARN.

tStatCatcher Statistics

Select this check box to gather the Job processing metadata at the Job level as well as at each component level.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage rule

This component is always used to start a Pig process and needs tPigStoreResult at the end to output its data.

In the Map/Reduce mode, you need only configure the Hadoop connection for the first tPigLoad component of a Pig process (a subJob), and any other tPigLoad component used in this process reuses automatically that connection created by that first tPigLoad component.

Prerequisites

The Hadoop distribution must be properly installed, so as to guarantee the interaction with Talend Studio . The following list presents MapR related information for example.

• Ensure that you have installed the MapR client in the machine where the Studio is, and added the MapR client library to the PATH variable of that machine. According to MapR's documentation, the library or libraries of a MapR client corresponding to each OS version can be found under MAPR_INSTALL\ hadoop\hadoop-VERSION\lib\native. For example, the library for Windows is \lib\native\MapRClient.dll in the MapR client jar file. For further information, see the following link from MapR: http://www.mapr.com/blog/basic-notes-on-configuring-eclipse-as-a-hadoop-development-environment-for-mapr.

  Without adding the specified library or libraries, you may encounter the following error: no MapRClient in java.library.path.

• Set the -Djava.library.path argument, for example, in the Job Run VM arguments area of the Run/Debug view in the Preferences dialog box in the Window menu. This argument provides to the Studio the path to the native library of that MapR client. This allows the subscription-based users to make full use of the Data viewer to view locally in the Studio the data stored in MapR.

For further information about how to install a Hadoop distribution, see the manuals corresponding to the Hadoop distribution you are using.

Limitation

Knowledge of Pig scripts is required. If you select HCatLoader as the load function, knowledge of HCataLog DDL(HCataLog Data Definition Language, a subset of Hive Data Definition Language) is required. For further information about HCataLog DDL, see https://cwiki.apache.org/confluence/display/Hive/HCatalog.