Limitations and considerations

The limitations and considerations differ according to the loading method you selected in the endpoint settings.

Limitations and considerations when using the Batch Loading loading method

The following limitations apply when using the Batch Loading loading method:

• The following DDLs are not supported:
  • Drop column
  • Rename column
  • Change column data type

• Capturing changes from source tables without a Primary Key or Unique Index is not supported. If you need to capture changes from such tables, you can add a Primary Key using a transformation. Additionally, Primary Key or Unique Index columns cannot contain NULLs. If you are aware that such columns will be populated with NULL values, define a transformation to change the NULLs to non-nullable values.

• JSON fields are not supported as primary keys.

• The Transactional apply Change Processing Mode is not supported.
• The Allow unlimited LOB size LOB handling option is not supported.

• With the following task settings configuration, the replication task will perform a DELETE operation followed by an INSERT instead of an UPDATE:

  • Change Processing Mode is set to Batch optimized apply.

  • The Apply changes using SQL MERGE option is not selected.

  • UPDATEs error handling for Apply Conflicts is set to: No record found for applying an UPDATE: INSERT the missing target record.

  As Google Cloud BigQuery does not support rollback, in the event that the replication task is unable to insert the updated row, the data will be deleted from the target.

Limitations and considerations when using the Streaming loading method

The following limitations apply when using the Streaming loading method:

• The following DDLs are not supported:
  • Drop column
  • Rename column
  • Change column data type

• Capturing changes from source tables without a Primary Key or Unique Index is not supported. If you need to capture changes from such tables, you can add a Primary Key using a transformation. Additionally, Primary Key or Unique Index columns cannot contain NULLs. If you are aware that such columns will be populated with NULL values, define a transformation to change the NULLs to non-nullable values. If the Control Tables are created with the Streaming loading method (see the second bullet above), Replicate will set each Control Table's Primary Key automatically, even if they already exist on the target.

• JSON fields are not supported as primary keys.

• The Allow unlimited LOB size LOB handling option is not supported.
• The Batch optimized apply Change Processing Mode is not supported.
• Switching from the Streaming loading method to Batch Loading requires you to wait approximately 90 minutes before performing DML operations on the source tables. The alternative is to reload all of the tables.
• The number of columns per table cannot exceed 2,000.
• Each record cannot exceed 10 MB.

• If you experience excessive latency during CDC, try setting the max_staleness option on tables with high latency.

  For more information, see Manage table staleness.

• By default, Control Tables on which UPDATE/DELETE operations are performed will be loaded using the Batch Loading method, even if the endpoint is set to use the Streaming method. This is because Control Tables are common to all tasks. However, if you know for certain that all your tasks are using the Streaming loading method, you can set a feature flag that will cause records to be streamed into the Control Tables instead. For more information, contact Qlik Support.
• Tables that use streaming with CDC, cannot run MERGE/UPDATE/DELETE statements immediately, it can take up to 90 minutes for BigQuery to unlock the table from UPSERT mode. That means an error would occur when switching from streaming after CDC to batch loading with the same table, and running one of these DMLs.
• The Suspended tables (attrep_suspended_tables) Control Table is not supported.
• When a table and is dropped from the target and then created by Replicate with the same name, the new table might not be created instantaneously. That can also occur when the replication task is configured to drop and create existing tables during Full Load.
• Apply Conflicts error-handling policies are not relevant as the Streaming loading method uses UPSERT for INSERT and UPDATE operations.

• The STATEMENT format of errors in the Apply exceptions (attrep_apply_exceptions) Control Table will be as follows:

  {"streamPosition":STREAM_POSITION,"operation":OPERATION}

  Where the OPERATION values can be INSERT, UPDATE or DELETE.