Troubleshooting data tasks

This section describes problems that can occur when working with data tasks and how to troubleshoot.

Troubleshooting environmental errors

When a data task encounters an environmental error, for example, timeouts, network errors, or connection error, the data task will retry the operation automatically. If the error is not resolved after retrying, the data task stops running, and shows the status Error with an error message.

• Landing tasks with data sources that are only accessible via Data Movement gateway:

  The operation is retried an infinite number of times, with an interval of 5 seconds.

  If the outage is long, the interval is doubled until an interval of 1800 seconds is reached.

• Landing tasks with data sources that are accessible without Data Movement gateway, Storage tasks, Transform tasks and Data mart tasks:

  The operation is retried 3 times, with an interval of 1 second.

Do the following:

1. Resolve the error using the error message.

2. Reload or resume operation of the data task.

Troubleshooting issues with a specific table

When a data task encounters an error while writing to a specific table, the data task will continue running. The table in error will show the status Error with an error message.

1. Resolve the error using the error message.

2. Reload the table that was in error.

Troubleshooting CDC issues

Landing data tasks with Full load & CDC update mode can encounter CDC related issues that affect the entire task, and that cannot be resolved by reloading specific tables. Examples of issues are missing events, issues caused by source database reorganization, or failure when reading source database events.

You can reload all tables to the target to resolve such issues.

1. Stop the data task and all tasks that consume it.

2. Open the data task and select the Monitor tab.

3. Click ..., and then Reload target.

This will reload all tables to the target using Drop-Create, and will restart all change data capture from now.

• Storage tasks that consume the landing data task will be reloaded via compare and apply at their next run to get in sync. Existing history will be kept. Type 2 history will be updated to reflect changes after the reload and compare process is executed.

  The timestamp for the from date in the type 2 history will reflect the reload date, and not necessarily the date the change occurred in the source.

• Storage live views will not be reliable during the reload target operation, and until the storage is in sync. Storage will be fully synced when:

  • All tables are reloaded using compare and apply,

  • One cycle of changes is performed for each table.

For more information, see Reloading all tables to the target.

NULL values in primary key columns

You may receive an error message when executing a data task: Unknown execution error - NULL result in a non-nullable column.

Possible cause  

Columns used as a primary key must not contain NULL values, and should be non-nullable.

Proposed action  

In the source data task, add an expression that converts all NULL values to a value, for example, 0.

You can also select another column to use as primary key.

Casting error when using Redshift as data platform

You may get the following error or similar when using Redshift as data platform: Failed to find conversion function from “unknown” to character varying

Possible cause  

Missing casting of a constant expression. This may happen more frequently in data marts due to the higher complexity of the final query .

Proposed action  

Cast the constant expression as text.

Example:

Ambiguous column names

When you register data based on a view created in a Qlik Talend Data Integration pipeline, the view may contain columns that were generated by Qlik Talend Data Integration. The names of these columns, starting with hdr__, are reserved. When a column with a reserved name is consumed in a storage task, the storage task will create columns with the same reserved name, leading to a naming conflict. For example, you can have two columns named hdr__key_hash.

For more information about reserved columns name in views, see Views.

Proposed action  

Rename the column that comes from the registered data task in the storage data task. For example, rename hdr__key_hash to my__key_hash.

Transformations are not applied to existing data

You have added or changed transformations in a task with existing data, but when you prepare and run the task again, the new or changed transformations are only applied to new data.

Proposed action  

Reload all affected tables if you want to apply the new or changed transformations to all existing data.