Change processing tuning

Transaction offload tuning

The following tuning options are available, regardless of which Change processing mode is selected:

• Offload transaction in progress to disk if:

  Qlik Talend Data Integration usually keeps transaction data in memory until it is fully committed to the source and/or target. However, transactions that are larger than the allocated memory or that are not committed within the specified time limit will be offloaded to disk.

  • Transaction memory size exceeds (MB): The maximum size that all transactions can occupy in memory before being offloaded to disk. The default value is 1024.
  • Transaction duration exceeds (seconds): The maximum time that each transaction can stay in memory before being offloaded to disk. The duration is calculated from the time that Qlik Talend Data Integration started capturing the transaction. The default value is 60.

Miscellaneous tuning

• Statements cache size (number of statements): The maximum number of prepared statements to store on the server for later execution (when applying changes to the target). The default is 50. The maximum is 200.

• Store task recovery data in target database: Select this option to store task-specific recovery information in the target database. When this option is selected, Qlik Talend Data Integration creates a table named attrep_txn_state in the target database. This table contains transaction data that can be used to recover a task in the event that the files in the Data Movement gateway Data folder are corrupted or if the storage device containing the Data folder has failed.

Batch optimized apply

The following settings are only available when the Apply mode is set to Batch optimized only.

• Apply batched changes in intervals:

• Larger than: The minimum amount of time to wait between each application of batch changes. The default value is 1.

  Increasing the Longer than value decreases the frequency with which changes are applied to the target while increasing the size of the batches. This can improve performance when applying changes to target databases that are optimized for processing large batches.

• Less than: The maximum amount of time to wait between each application of batch changes (before declaring a timeout). In other words, the maximum acceptable latency. The default value is 30. This value determines the maximum amount of time to wait before applying the changes, after the Longer than value has been reached.

• Force apply a batch when processing memory exceeds (MB): The maximum amount of memory to use for pre-processing in Batch optimized apply mode. The default value is 500.

  For maximum batch size, set this value to the highest amount of memory you can allocate to Qlik Talend Data Integration. This can improve performance when applying changes to target databases that are optimized for processing large batches.

Transactional apply

The following settings are only available when the Apply mode is set to Transactional.

• Minimum number of changes per transaction: The minimum number of changes to include in each transaction. The default value is 1000.

  Information noteQlik Talend Data Integration applies the changes to the target either when the number of changes is equal to or greater than the Minimum number of changes per transaction value OR when the batch timeout value is reached (see below) - whichever occurs first. Because the frequency of changes applied to the target is controlled by these two parameters, changes to the source records may not immediately be reflected in the target records.
• Maximum time to batch transactions before applying (seconds): The maximum time to collect transactions in batches before declaring a timeout. The default value is 1.