Skip to main content

Creating a Change Processing ETL Task

ON THIS PAGE

Creating a Change Processing ETL Task

You can create a CDC ETL set by duplicating the Full Load ETL set. This is useful if you currently have a Full Load ETL set only and the need (which wasn't previously anticipated) arises to capture changes from all of the tables or only from certain tables.

  1. Select the Full Load ETL Set in the left pane of the Manage ETL Sets window and click the Duplicate toolbar button.
  2. In the Duplicate <ETL_Set_Name> window, set the following properties:

    • ETL name - The name for the ETL set (e.g. employees_changes).
    • Select a landing area - Select the same landing area as the Full Load ETL set.
    • Schemas - Select the same schema as the Full Load ETL set.
    • ETL type - Select Change Tables.
  3. To only apply changes to selected tables, in the Mappings column, select the tables whose changes you want to apply to the data warehouse.

    Note:

    If you want to modify a column's mapping, the table name displayed in the left pane of the Edit Mappings window will not be appended with the "__ct" suffix (which is the default naming format for Replicate Change Tables). However, the changes will still be taken from the associated Change Table.

    Warning:

    For each Primary Key there is one record. Changing a Primary Key on the source record will cause a new record to be inserted in the data warehouse.

  4. Click Generate and wait for the ETL instructions to be generated.
  5. To capture the changes and apply them to the data warehouse, click Run.

Reloading data from the source tables to the Landing Area

In cases of inconsistencies or when metadata from the source tables is not replicated to the Landing Area, you may need to reload the data.

  1. Run the Qlik Replicate Full Load replication task again.
  2. Run the Compose for Data Warehouses Full Load storage task.

  3. Continue running the existing Compose Change Processing task.

    Note:

    No duplicates are created, as Compose compares the records before adding them to the Data Warehouse.

    Warning:

    Records that should be deleted in the new Full Load task are not deleted nor marked for deletion (soft delete).