Skip to main content Skip to complementary content

tKuduOutput properties for Apache Spark Batch

These properties are used to configure tKuduOutput running in the Spark Batch Job framework.

The Spark Batch tKuduOutput component belongs to the Databases family.

The component in this framework is available in all Talend products with Big Data and Talend Data Fabric.

Basic settings

Use an existing configuration

Select this check box and in the Component List drop-down list, select the desired connection component to reuse the connection details you already defined.

Server connection

Click the [+] button to add as many rows as the Kudu masters you need to use, each row for a master.

Then enter the locations and the listening ports of the master nodes of the Kudu service to be used.

This component supports only the Apache Kudu service installed on Cloudera.

For compatibility information between Apache Kudu and Cloudera, see the related Cloudera documentation:Compatibility Matrix for Apache Kudu.

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

  • Built-In: You create and store the schema locally for this component only.

  • Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

 
Information noteNote: The schema of a Kudu table must declare a primary key comprised of one or more columns. These columns must be non-nullable, and may not be of type boolean, float or double.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

Kudu table

Enter the name of the table to be created, changed or removed.

Action on table

Select an operation to be performed on the table defined.

  • None: No operation is carried out.

  • Drop and create table: The table is removed and created again.

  • Create table: The table does not exist and gets created.

  • Create table if does not exist: The table is created if it does not exist.

  • Drop table if exist and create: The table is removed if it already exists and created again.

Action on data

Select an action to be performed on data of the table defined.

  • Insert: Add new entries to the table. If duplicates are found, the Job stops.

  • Update: Make changes to existing entries.

  • Upsert: Update the record with the given reference. If the record does not exist, a new record would be inserted.

  • Delete: Remove entries corresponding to the input flow.

Replicas

Enter, without double quotation marks, the replication factor of this table to create copies of your table and its tablets.

For further information about Kudu tablets and Kudu replication policies, see Distribution and Fault Tolerance.

Hash partitions

When you are creating a Kudu table, it is recommended to define how this table is partitioned. By default, your table is not partitioned.

  1. Click the [+] button to add the columns based on which rows are hash-partitioned, for example, add a host column and a metric column.

    These columns must exist in your data and in the schema you have defined in tKuduOutput.

  2. In Number of buckets, enter, without double quotation marks, the number of the buckets to be used to store the partitions. These buckets are created on the fly.

At runtime, rows are distributed by hash value in one of those buckets. If you leave this Hash partitions table empty, hash partitioning is not applied during the creation of the table.

For further information about hash partitioning in Kudu, see Hash partitioning.

Range partitions

When you are creating a Kudu table, it is recommended to define how this table is partitioned. By default, your table is not partitioned.

  1. Click the [+] button to add the primary key columns based on which rows are partitioned to contiguous segments, for example, add a time column.

    These columns must exist in your data and in the schema you have defined in tKuduOutput.

  2. In N. Partition, enter, without double quotation marks, a number to be used as the ID number of the partition to be created. For example, enter 1 to create Partition 1.
  3. In Lower boundary and Upper boundary, enter the boundaries between which rows are partitioned. The upper boundary is excluded and only the data between the boundaries (including the lower boundary) is written to the Kudu table.

    For example, if you are partitioning your data based on a time column of Date type and the format of your time data is yyyy-mm-dd, you can set the lower boundary to be 2016-01-01 by entering, without double quotation marks, TalendDate.parseDate("yyyy-MM-dd", "2016-01-01") and do the same to set the upper boundary to be 2018-01-01, then only rows with dates between the two boundaries enter this partition.

At runtime, rows of these columns are distributed using the values of the columns you have added to this Range partitions table. If you leave this table empty, range partitioning is not applied during the creation of the table.

For further information about hash partitioning in Kudu, see Range partitioning.

Die on error

Select the check box to stop the execution of the Job when an error occurs.

Usage

Usage rule

This component is used as an end component and requires an input link.

Spark Connection

In the Spark Configuration tab in the Run view, define the connection to a given Spark cluster for the whole Job. In addition, since the Job expects its dependent jar files for execution, you must specify the directory in the file system to which these jar files are transferred so that Spark can access these files:
  • Yarn mode (Yarn client or Yarn cluster):
    • When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab.

    • When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab.

    • When using Altus, specify the S3 bucket or the Azure Data Lake Storage for Job deployment in the Spark configuration tab.
    • When using Qubole, add a tS3Configuration to your Job to write your actual business data in the S3 system with Qubole. Without tS3Configuration, this business data is written in the Qubole HDFS system and destroyed once you shut down your cluster.
    • When using on-premises distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration.

  • Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration Apache Spark Batch or tS3Configuration Apache Spark Batch.

    If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem).

This connection is effective on a per-Job basis.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!