Doing continuous matching using tMatchIndexPredict
This scenario applies only to subscription-based Talend Platform products with Big Data and Talend Data Fabric.
After indexing lookup data in Elasticsearch using tMatchIndex, you do not need to restart the matching process from scratch. The tMatchIndexPredict component compares new data records with the lookup stored in ElasticSearch.
In this example, a list of early childhood education centers in Chicago coming from ten different source has been cleaned, deduplicated and indexed in Elasticsearch. You want to match new records which contain information about early childhood education centers in Chicago against the reference data set stored in Elasticsearch.
tMatchIndexPredict uses pairing and matching models to group together records from the input data and the matching records from the reference data set indexed in Elasticsearch and label the suspect pairs.
tMatchIndexPredict outputs potential duplicates and unique records in separate files.
-
You generated a pairing model.
For examples of how to generate a pairing model, see Computing suspect pairs and suspect sample from source data and Computing suspect pairs and writing a sample in .
-
You generated a matching model.
For examples of how to generate a matching model, see Generating a matching model and Generating a matching model from a Grouping campaign.
-
Clean and deduplicated data has been indexed in Elasticsearch to match against new data records and determine whether they are unique records or suspect duplicates.
For further information, see Indexing a reference data set in Elasticsearch.
-
The Elasticsearch search cluster must be running ElasticSearch 5+.