BigData framework

This framework is equivalent to Staging, but it is adapted to big data targets (such as Snowflake or Netezza). Data transactions are processed in bulks (or batch works of data) according to a defined frequency or after reaching a defined number of records. First, source data are mirrored in the target table. Then, DD_ID and DD_ACTION are added to this target table, so that all updates can be displayed.

1. Install your target engine. For more details, go to Installing the target engines section.
2. Create an account on Snowflake.
3. Follow Snowflake requirements section.
4. From the Manager, subscribe to a source engine.
5. Create an environment, then a model.
6. From the window called Adding tables to the model, select BigData.
7. Select the table(s) and click Ok.
8. Create a target. From the New Target window, enter the Name and IP address of the target machine.
9. Distribute the model. In the Distribution window, set the parameters as explained in section https://help.talend.com/r/en-US/7.3/change-data-capture-installation-guide/installing-snowflake-connector.
10. Follow Preparing the replication procedure starting from step 7, then Operating a replication.