tMatchIndexPredict

Compares a new data set with a lookup data set stored in ElasticSearch, using tMatchIndex. tMatchIndexPredict outputs unique records and suspect duplicates in separate files.

In the potential duplicates output, each record contains the fields from the source records and the fields from the potentially matching lookup records.

The tMatchIndexPredict component supports Elasticsearch versions up to 6.4.2 and Apache Spark versions 2.0, 2.1, 2.2, 2.3 and 2.4 in local mode.

As this component does not support the Elasticsearch authentication, it cannot run on Databricks.