tIcebergOutput properties for Apache Spark Batch
These properties are used to configure tIcebergOutput running in the Spark Batch Job framework.
The Spark Batch tIcebergOutput component belongs to the Big Data family.
The component in this framework is available in all Talend products.
This component is available when you have installed the 8.0.1-R2024-04 Studio Monthly or a later one provided by Talend. For more information, check with your administrator.
Basic settings
| Properties | Description |
|---|---|
| Catalog and Schema | To retrieve a dataset from Unity Catalog, specify the following parameters:
These fields are available only when you select Enable Unity Catalog with Databricks in the Spark Configuration tab of your Spark Batch Job. |
| Table | Enter the name of the table in which you want to write data into. |
| Action on table | Select the action you want to perform on the table from the drop-down
list:
|
| Set format | Select this check box to specify the format, and then select the format
information from the Format version and File
format drop-down lists. This parameter is only available when you select Create, Create or replace, or Replace from the Action on table drop-down list. This parameter is not available when you select Enable Unity Catalog with Databricks in the Spark Configuration tab of your Spark Batch Job. |
| Set location | Select this check box to specify a location path in which you want the table
to be created. This parameter is only available when you select Create, Create or replace, or Replace from the Action on table drop-down list. |
| Set partitions | Select this check box to create partitions inside the table, and then
specify the name of the columns in the Partition key table
that appears. This parameter is only available when you select Create, Create or replace, or Replace from the Action on table drop-down list. |
| Table properties | Enter any extra configuration properties following the key and values format
by clicking the plus button. For example, you can add properties such as
format-version. For more information on available
configuration properties, see Configuration from Apache Iceberg documentation. This parameter is only available when you select Create, Create or replace, or Replace from the Action on table drop-down list. |
Global Variables
| Variables | Description |
|---|---|
|
ERROR_MESSAGE |
The error message generated by the component when an error occurs. This is an After variable and it returns a string. |
Usage
| Usage guidance | Description |
|---|---|
| Usage rule | This component is generally used with other Iceberg components. |