Change Data Capture (CDC)

The CDC operator full-joins records from 2 inputs (a primary and secondary entity) based on user-specified join key or keys.

The operator compares the fields specified through user-specified compare criteria.

From this comparison; (I) Inserted, (U) Updated, (D) Deleted records /(N) No-change /(E) Error records are classified in output (through Record Output setting) as either:

Changed Only (default): Output will contain a union of all records (I) Inserted /(U) Updated /(D) Deleted between two source entities as indicated in the IUD field

Full: Output will contain a union of all records (I) Inserted / (U) Updated / (D) Deleted /(N) No-change between two source entities as indicated in the IUD field

This target entity contains a union of all records and has an IUD field (podium_iud), which indicates fields' change status.

Application of CDC package

To apply the CDC operator, select and add 2 entities (a primary baseline and the secondary entity that will provide the change data) onto the canvas

Once the CDC package is dragged to the canvas and connected to entities, define CDC operator settings.

CDC requirements

CDC package expects two inputs (one for primary and one for the newer secondary table) and both entities should have the same record format (name, data type, index).

The order in which the ports are connected has an impact on the functionality. The secondary entity added to the graph (should also be the "changed" entity) should be connected first followed by the primary package.

CDC operator must always be the last component in a dataflow (inputs to the CDC must be last in the sequence of other controls; Filter, Router, Transform, etc.) for dataflow transformations to factor into the calculation,

CDC operator Settings

• ADD CRITERIA:Specify fields (join keys) for Full Join or Swap where the right entity takes on join primacy and data change is captured in this direction from secondary entity to primary entity.

COMPARE

• ADD COMPARE: Add and specify fields for comparison between entities.  If Compare criteria are empty, all fields except for the join keys are compared.

SETTING

• Log Changes: If checked (default), the output will create two sub-folders in the receiving directory of the master entity. One '/good' folder will contain the output data as usual and the '/log' folder will contain the log of changes, if any. If unchecked, no log will be created.

 

Logging output will list compare key and all columns (fields) with the updated fields showing old and new values if change occurred:

Record output settings:

• Changed only (default): Target will contain only DUI records
• Full: Target will contain DUIN records (a full set)
• D: Delete
• U: Update (change in existing record)
• I: Insert (new records)
• N: Unchanged

Define and Add Target Entity

Save, Validate, and Execute the dataflow.

Behavior of CDC logs

• Logs will be saved in the first entity if multiple targets are selected through Router.
• Logs will not be filtered if the Filter is applied after the CDC control.
• Logs will be filtered if the Filter is applied before the CDC control.