How does Talend Change Data Capture work?

The Talend Change Data Capture engine retrieves the changes made to the database from the transaction journals of the source machine. The use of the incremental mode allows it to operate in real time (on the fly) and considerably reduces the volume of data to refresh. For instance, your eBusiness system that informs your customers about the status of their order is updated online. At night, your decision support system receives only new and changed data, which greatly reduces the volumes to be transferred. The incremental mode can, for example, be implemented according to a set schedule during night hours to refresh your datawarehouse.

With Talend Change Data Capture, you can also work directly with source files or tables without requiring transaction journal support. With the EXTRACT mode, you can perform a complete refresh with or without selection (based on a date, for example).

For each transaction record processed by Talend Change Data Capture, the transformation module offers unlimited possibilities to select and enhance the data by calling customized programs where necessary. The selection/transformation is performed on the source machine so that only essential data in the appropriate form is sent to the target machine, thus greatly curbing the volumes exchanged. The selection/transformation is also performed on the target machine so you can choose where you want to consume power.

The data is transmitted in blocks and processed in parallel on the target machine. This way, the target machine processes a block while the next block is prepared on the source machine. This ensures that no time is lost and that large volumes are absorbed smoothly. On receiving the block, the target machine applies it to the target database using sophisticated algorithms that integrate error management with an automatic repair feature the user can select.

Given its ability to call a customized program on the target machine, it can interface with an ERP software package, for example, and call its APIs rather than write directly to its database.

Talend Change Data Capture can load a target database whose structure is radically different from that of the source, in real time. You can define record merges and splits in the parameters.

Talend Change Data Capture also offers the unique possibility of knowing and working with data changes. For example, you can know the exact change in price, in terms of the amount or the percentage as soon as the change is applied in the transformation module on the source machine. In short, instead of selecting a record for which the code X contains the value ‘B’, you can do so based on whether the code X has changed from value ‘A’ to value ‘B’. You can also send the former price to the target machine along with the new price in order to update an aggregate simultaneously (in real time) on the target machine.

Talend Change Data Capture can be started and stopped easily from your applications and your administration tools. The Talend Change Data Capture alerts are standardized so that they can be easily integrated in your alert upload tools. In AS/400 IBMi, Talend Change Data Capture integrates a module used to automate journal receiver cleanup, thus optimizing the disk space used.