Replication mapping for Emulating a Database Replication / DI / ETL / ELT / CDC Tool

Some DI tools specify data movement logic at the table (or even schema) level. A database replication tool can ask to load a table from a file without specifying columns involved. TDC customers would like to trace lineage through these bulk metadata at the column level.

The TDC metamodel and Database/DI bridges support classifier-level-only mappings. The data mapper supports authoring of these type of mappings and resolves these not completely defined and Mapper mappings during the configuration assembly (verification) process.

Use cases

Database replication tools

Database replication tools, like Oracle GoldenGate and HVR allows you to synchronize database at catalog, schema, table and column level. They support inclusion and exclusion rules, like all but these tables in this schema. The rules can use wildcards and regular expressions. They can use conditions, like where a>1.

Database file import/export, like Oracle bulk loader

To use a database bulk loader, you need a control file, which specifies how data should be loaded into the database; and a data file, which specifies what data should be loaded.

Apache Sqoop

Apache Sqoop efficiently transfers bulk data between Apache Hadoop (HDFS/Hive) and structured datastores such as relational databases. Sqoop can also be used to extract data from Hadoop and export it into external structured datastores.

MM Mapping Specification

When users cannot import replication mapping metadata they would like to specify it manually in TDC.

Detect link algorithm

One may specify prefix and suffix for a replication mapping. These are not filters but rather matching helpers. For example, E.g., if all the names in the Target begin with WWX_ (WWX_NAME, WWX_ADDRESS, WWX_DOB, etc.), then you would place WWX_ in Target name > Add Prefix.