Full load
Basic
If target table already exists: Select one of the following from the list to determine how you want to handle loading the target at full-load start up:
The option to drop or truncate the target tables is relevant only if such operations are supported by the source endpoint.
-
Drop and create table: The table is dropped and a new table is created in its place.
Information noteQlik Talend Data Integration Control Tables will not be dropped. However, any suspended tables that are dropped will also be deleted from the attrep_suspended_tables Control Table if the associated task is reloaded.
- TRUNCATE before loading: Data is truncated without affecting the table metadata. Note that when this option is selected, enabling the Create primary key or unique index after full load completes option will have no effect.
- Ignore: Existing data and metadata of the target table will not be affected. New data will be added to the table.
Advanced
Performance tuning
The following performance tuning parameters can be set:
- Maximum number of tables: Enter the maximum number of tables to load into the target at one time. The default value is 5.
-
Transaction consistency timeout (seconds): Enter the number of seconds that Qlik Talend Data Integration waits for transactions to close, if they are open when the task starts, before beginning the Full Load operation. The default value is 600 (10 minutes). Qlik Talend Data Integration will begin the full load after the timeout value is reached even if there are open transactions.
Information noteTo replicate transactions that were open when Full Load started but were only committed after the timeout value was reached, you need to reload the target tables. - Commit rate during full load: The maximum number of events that can be transferred together. The default value is 10000.
After full load completes
You can set the task to stop automatically after Full Load completes. This is useful if you need to perform DBA operations on the target tables before the task’s Apply Changes (i.e. CDC) phase begins.
During Full Load, any DML operations executed on the source tables are cached. When Full Load completes, the cached changes are automatically applied to the target tables (as long as the Before/After cached changes are applied options described below are disabled).
- Create primary key or unique index: Select this option if you want to delay primary key or unique index creation on the target until after full load completes.
- Stop the task:
Information noteThese settings are not available when replicating from SaaS application sources as there are no cached events.
-
Before cached changes are applied: Select to stop the task after Full Load completes.
-
After cached changes are applied: Select to stop the task as soon as data is consistent across all tables in the task.
Information noteWhen configuring the task to be stopped after Full Load completes, note the following:
- The task does not stop the moment Full Load completes. It will be stopped only after the first batch of changes has been captured (as this is what triggers the task to stop). This might take a while depending on how frequently the source database is updated. After the task stops, the changes will not be applied to the target until the task is resumed.
- Choosing Before cached changes are applied might impact performance, as the cached changes will only be applied to tables (even those that have already completed Full Load) after the last table completes Full Load.
- When this option is selected and a DDL is executed on one of the source tables during the Full Load process (in a Full Load and Apply Changes task), Qlik Talend Data Integration will reload the table. This effectively means that any DML operations executed on the source tables will be replicated to the target before the task stops.
-