Data pipeline project settings

You can change the settings for a data pipeline project in Qlik Talend Data Integration. The properties are common to the project and all included data tasks. Some settings are only available for specific data platforms.

• Click Settings in the project.

Data platform

You can change the following settings:

• Connection

  Connection for the project.

• Connection to staging area

  This option is not available when the data platform is Snowflake.

Metadata

You can set a suffix for internal artifacts and default suffixes for views that are created.

• Internal artifacts preferences

  Set the suffix to be used for schemas used to store internal artifacts in Schema suffix.

• External views suffixes

  Set default suffixes for views that are created in data tasks included in the project.

Default settings for new tasks

You can set default values for data tasks that are created in the project. When you create a data task you can change the value.

You can set the default database to create target artifacts for all types of data tasks.

Landing default settings

• Proxy server when using Data Movement gateway

  You can select to use a proxy server when the Data Movement gateway connects to the cloud data warehouse and the storage area.

  For more information about configuring the Data Movement gateway to use a proxy server, see Setting the Qlik Cloud tenant and a proxy server.

  • Use proxy to connect to cloud data warehouse

    Information noteAvailable when using Snowflake, Google BigQuery, and Databricks.

  • Use proxy to connect to storage

    Information noteAvailable when using Azure Synapse Analytics, Amazon Redshift, and Databricks.

Storage default settings

• Historical Data Store (Type 2)

  You can keep historical change data to let you easily recreate data as it looked at a specific point in time. You can use history views and live history views to see the historical data.

• Live views

  Live views show a view for each selected source table which merges the table with changes from the change table. This provides queries with a live view of the data without having to wait for the next apply cycle.

Default database

You can use the default database of the project or specify another database.

Catalog

• Publish to catalog

  Select this option to publish this version of the data to Catalog as a dataset. The Catalog content will be updated the next time you prepare this task.

Transform default settings

• Historical Data Store (Type 2)

  You can keep historical change data to let you easily recreate data as it looked at a specific point in time. You can use history views and live history views to see the historical data.

Materialization

• Non-materialized (Views only)

  Select this option to only create views that perform transformations on the fly.

• Materialized (Tables and Views)

  Select this option to create both tables and views.

Default database

You can use the default database of the project or specify another database.

Catalog

• Publish to catalog

  Select this option to publish this version of the data to Catalog as a dataset. The Catalog content will be updated the next time you prepare this task.

Data mart default settings

Default database

You can use the default database of the project or specify another database.

Catalog

• Publish to catalog

  Select this option to publish this version of the data to Catalog as a dataset. The Catalog content will be updated the next time you prepare this task.

Registered data default settings

Default database

You can use the default database of the project or specify another database.

Catalog

• Publish to catalog

  Select this option to publish this version of the data to Catalog as a dataset. The Catalog content will be updated the next time you prepare this task.

Incremental load settings

These settings are available when Incremental using high watermark is selected.

• Change tables

  If the changes are in the same table, select Changes are within the same table.

  If not, deselect Changes are within the same table and specify a change table pattern.

• Watermark column

  Set the name of the watermark column in Name.

• "From date" column

  You can indicate the "From date" by the start time, or using a selected column.

  If you select Selected "From date" column, you must define a "From date" pattern.

• Soft deletions

  You can include soft deletions in changes by selecting Changes include soft deletions and defining an indication expression.

  The indication expression should evaluate to True if the change is a soft delete.

  Example: ${is_deleted} = 1

• Before image

  You can filter out before image records in change tables changes by selecting Before image and defining an indication expression.

  The indication expression should evaluate to True if the row contains the image before the update.

  Example: ${header__change_oper} = 'B'

Runtime

You can define default runtime performance settings for data assets that are included in the project.

Landing default values

• You can set the maximum number of database connections in Parallel execution.

Storage default values

• You can set default scheduling settings to a time based schedule. This will be the default value for each storage task created.

• You can set the default data warehouse if the project platform is Snowflake.

Transform default values

• You can set default scheduling settings to a time based schedule or On successful completion of any input data task. This will be the default value for each transformation task created.

• You can set the default data warehouse if the project platform is Snowflake.

Data mart default values

• You can set default scheduling settings to a time based schedule or On successful completion of any input data task. This will be the default value for each data mart task created.

• You can set the default data warehouse if the project platform is Snowflake.