# tMatchGroup properties for Apache Spark Batch

These properties are used to configure tMatchGroup running in the Spark Batch Job framework.

The Spark Batch tMatchGroup component belongs to the Data Quality family.

This component is available from the 8.0 R2024-08 Talend Studio monthly release.

Information noteImportant: When using the Simple VSR algorithm, the results depend on the input data and the Spark API shuffling. This causes different results at each Job run. To optimize the result stability, it is highly recommended to use the [tPartition component](https://help.qlik.com/talend/en-US/components/8.0/processing/tpartition) right before the tMatchGroup component.

To optimize the result stability, you can also:

* Set the same Blocking key (tMatchGroup) and Partition key (tPartition).

* Select unique values as the Sort key (tPartition). For example, the ID.

The component in this framework is available in all Talend Platform products with Big Data and in Talend Data Fabric.

## Basic settings

| Properties | Description |
| --- | --- |
| Schema and Edit schema | <ul> <li> <p>A schema is a row description. It defines the number of fields            (columns) to be processed and passed on to the next component. When you create a Spark            Job, avoid the reserved word <kbd>line</kbd> when naming the            fields.</p> <p>Click <span>Sync        columns</span> to retrieve the schema from the previous component connected in the      Job.</p> <p>The output schema of this component contains the following                     read-only fields:</p> <p>The output schema of this component contains                        the following read-only fields:</p> <div> <ul> <li> <p> <span><samp>GID</samp>: Provides a group                        identifier of the data type <em>String</em>. </span> </p> </li> <li> <p> <span><samp>GRP_SIZE</samp>: Counts the                        number of records in the group, computed only on the master record.</span> </p> </li> <li> <p> <span><samp>MASTER</samp>: Identifies, by                            <span>true</span> or <span>false</span>, if the record used in the matching comparisons is a                        master record. There is only one master record per group. Each input record                        will be compared to the master record, if they match, the input record will                        be in the group.</span> </p> </li> <li> <p> <span><samp>SCORE</samp>: Measures the                        distance between the input record and the master record according to the                        matching algorithm used. In case the <span>tMatchGroup</span> component is used to have multiple output                        flows, the score in this column decides to what output group the record                        should go.</span> </p> </li> <li> <p> <span><samp>GRP_QUALITY</samp> provides the                                    quality of similarities in the group by taking the minimal                                    matching value. Only the master record has a quality                                score.</span> </p> </li> </ul> </div> </li> <li> <p> <span>Built-In</span>: You create and store the schema locally for this component                only.</p> </li> <li> <p> <span>Repository</span>: You have already created the schema and stored it in the                Repository. You can reuse it in various projects and Job designs.</p> </li> </ul> |
| Matching<br>Algorithm | Simple VSR is set by<br>default.<br><br>Information noteRestriction: The T-Swoosh algorithm is not available for<br>the Apache Spark Batch component. You can use it for the [Standard<br>component](https://help.qlik.com/talend/en-US/data-matching/8.0/tmatchgroup-standard-properties "These properties are used to configure tMatchGroup running in the Standard Job framework."). |
| ![Import match rule](https://help.qlik.com/talend/en-US/data-matching/Resources/images/import_action.svg) | Click the import icon to import a match rule from the Talend Studio repository.<br><br>For further<br>information about how to import rules, see [Importing match rules from the<br>repository](https://help.qlik.com/talend/en-US/data-matching/8.0/importing-match-rules-from-studio-repository) |
| Key Definition - Input Key Attribute column | Select the columns from the input flow on which you want to<br>apply a matching algorithm.<br><br>Information noteNote: When you select a date column on which to apply an algorithm or a matching<br>algorithm, you can decide what to compare in the date format.<br><br>For example,<br>if you want to only compare the year in the date, in the component schema set the type<br>of the date column to Date and then enter "*yyyy*" in the Date Pattern field.<br>The component then converts the date format to a string according to the pattern defined<br>in the schema before starting a string comparison. |
| Key Definition - Matching<br>Function column | Select a matching algorithm from the list. For more<br>information on each algorithm, see [Matching algorithms](https://help.qlik.com/talend/en-US/data-matching/8.0/tmatchgroup-properties-for-apache-spark-batch#matching-algorithms-section). |
| Key Definition - Custom Matcher<br>Class column | When you select Custom as the matching<br>type, enter the path pointing to the custom class (external matching<br>algorithm) you need to use. This path is defined by yourself in the<br>library file (*.jar* file) which<br>you can import by using the tLibraryLoad component.<br><br>For example, to use a *MyDistance.class* class stored in the directory<br>*org/talend/mydistance* in a<br>user-defined *mydistance.jar*<br>library, the path to be entered is *org.talend.mydistance.MyDistance*. |
| Key Definition - Tokenized<br>measure column | Tokenization is the concept of splitting a string into<br>words. Select the method to use to compute a tokenized measure for the<br>selected algorithm:<br><br><ul> <li> <span>No</span>: No tokenization method is used on the                                string. With this option, "John Doe" and "Jon Doe" should                                match.</li> <li> <span>Same place</span>: Splits the two strings by words                                to two lists, list1 and list2. Associates each element from list1                                with the element which has the same position in list2. Using this                                method, "She is red and he is pink" and "Catherine is red and he is                                pink" should match.</li> <li> <span>Same order</span>: Splits the two strings by words                                to two lists, list1 and list2 and assumes that list1 is shorter than                                list2. Tries to associate the elements from list1 with the elements                                in list2 taken in the same order. Using this method, "John Doe" and                                "John B. Doe" match.</li> <li>This method should be used only with strings which has a few words,                                otherwise the number of possible combinations can be large.</li> <li> <span>Any order</span>: Splits the two strings by words to                                two lists, list1 and list2 and assumes that list1 is shorter than                                list2. Tries to assign each word of list1 to a word of list2, in                                order to have the highest global similarity (with respect to the                                used similarity). <p>Using this method, "John Doe" and "Doe John"                                    match.</p> </li> </ul> |
| Key Definition - Confidence<br>Weight column | Set a numerical weight for each attribute (column) of the key<br>definition.<br><br>You can enter a number or a context variable.<br><br>The value must be an integer greater than 0. |
| Key Definition - Handle<br>Null column | To handle null values, select from the<br>list the null operator you want to use on the column:<br><br><br>Null Match Null: A Null attribute<br>only matches another Null attribute.<br><br><br>Null Match NONE: A Null attribute<br>never matches another attribute.<br><br><br>Null Match ALL: A Null attribute<br>matches any other value of an attribute.<br><br>For example, two columns, *name* and *firstname* where the name is never null, but the first<br>name can be null.<br><br>If two records:<br><br>"Doe", "John"<br><br>"Doe", ""<br><br>Depending on the operator you choose, these two records<br>may or may not match:<br><br><br>Null Match Null: They do not<br>match.<br><br><br>Null Match NONE: They do not<br>match.<br><br><br>Null Match ALL: They match.<br><br>For the records:<br><br>"Doe", ""<br><br>"Doe", ""<br><br><br>Null Match Null: They match.<br><br><br>Null Match NONE: They do not<br>match.<br><br><br>Null Match ALL: They match. |
| Match<br>Threshold | Enter the match<br>probability. Two data records match when the probability is greater than<br>or equal to the set value.<br><br>You<br>can enter a different match threshold for each match rule. |
| Blocking Selection | If required, select the columns from the input flow according<br>to which you want to partition the processed data in blocks, this is usually<br>referred to as "blocking".<br><br>Blocking reduces the number of pairs of records that needs to<br>be examined. In blocking, input data is partitioned into exhaustive blocks<br>designed to increase the proportion of matches observed while decreasing the<br>number of pairs to compare. Comparisons are restricted to record pairs within<br>each block.<br><br>Using blocking columns is very useful when you are processing<br>very big data. |

## Advanced settings

| Properties | Description |
| --- | --- |
| Multiple output | Select the Separate output<br>check box to have several output flows:<br><br><ul> <li> <span>Uniques</span>: when the group size                                is equal to <em>1</em>, the record is listed in this                                flow. <p>When records are not unique, they can be:</p> </li> <li> <span>Matches</span>: when the                                                  group quality is greater than or equal to the threshold                                                  you define in the <span>Confident match                                                  threshold</span> field, the record is listed                                                  in this flow. </li> <li> <span>Suspects</span>: when the group quality                                            is less than the threshold you define in the <span>Confident match                                                threshold</span> field, the record is listed in                                            this flow.</li> </ul><br><br><br>Confident match<br>threshold: set a numerical value between the<br>current Match<br>threshold and *1*. From this threshold, you can be<br>confident in the quality of the group. |
| Multi-pass | Select this check<br>box to enable a tMatchGroup<br>component to receive data sets from another tMatchGroup that<br>precedes it in the Job. This will refine the groups received by each of<br>the tMatchGroup<br>components through creating data partitions based on different blocking<br>keys.<br><br>With multi-pass matching, all<br>master records are generated but intermediate master records are removed<br>from the output flow. Only final master and original records are kept at<br>the end.<br><br>When single master records from the 1st tMatchGroup merge<br>into one group after passing the 2nd tMatchGroup, their<br>order in the group may change every time you run the Job.<br><br>For an example Job, see [Matching customer data through multiple passes](https://help.qlik.com/talend/en-US/data-matching/8.0/matching-customer-data-through-multiple-passes). |
| Sort the output data by<br>GID | Select this check box to group the output data by the group<br>identifier.<br><br>The output is sorted in descending alphanumeric order by group identifier. |
| Output distance details | Select this check box to add an output column *MATCHING\_DISTANCES* in the schema of the<br>component. This column provides the distance between the input and<br>master records in each group.<br><br>Information noteImportant: When you use two tMatchGroup<br>components in a Job and select this option in one component, you must<br>select this check box in both tMatchGroup components<br>before linking them together. If you linked the components before<br>selecting this check box, select this check box in the second component<br>in the Job flow and then, in the first component. Otherwise, you may<br>have an issue as there are two columns in the output schema with the<br>same name. Selecting this check box in only one<br>tMatchGroup component may cause schema mismatch<br>issues. |
| Display detailed labels | This checkbox is available when Output distance details<br>is selected.<br><br>Select this check<br>box to have in the output *MATCHING\_DISTANCES* column not only the matching distance but<br>also the names of the columns used as key attributes in the applied<br>rule.<br><br>For example, if you try to<br>match on first name and last name fields, *lname* and *fname*, the output would be fname:1.0\|lname:0.97 when the check box<br>is selected and 1.0\|0.97 when it is not selected. |
| Deactivate matching computation when opening the<br>wizard | Select this check box to open the<br>Configuration wizard without running the match rules<br>defined in the wizard.<br><br>This<br>enables you to have a better experience with the component. Otherwise,<br>the wizard may take some time to open. |

## Global Variables

| Variables | Description |
| --- | --- |
| Global<br>Variables | ERROR\_MESSAGE: the error message generated by the component when an<br>error occurs. This is an After variable and it returns a string. This variable functions<br>only if the Die on error check box<br>is cleared, if the component has this check<br>box.<br><br>A Flow variable functions during the execution of a component<br>while an After variable functions after the execution of the component.<br><br>To fill up a field or expression with a variable, press Ctrl+Space to access the variable list and choose the variable to use from<br>it.<br><br>For more information about variables, see [Using contexts and variables](https://help.qlik.com/talend/r/en-US/8.0/studio-user-guide/contexts-and-variables). |

## Usage

| Usage guidance | Description |
| --- | --- |
| **Usage<br>rule** | This component is<br>an intermediary step. It requires an input flow as well as an output<br>flow. |

## Matching algorithms

This list describes each matching algorithm from the Matching Function column:

* Exact: Matches each processed entry to all possible reference entries with exactly the same value. It returns *1* when the two strings exactly match, otherwise it returns *0*.
* Exact - ignore case: Matches each processed entry to all possible reference entries with exactly the same value while ignoring the value case.
* Soundex: Matches processed entries according to a standard English phonetic algorithm. It indexes strings by sound, as pronounced in English, for example "Hello": "H400".It does not support Chinese characters.
* Levenshtein (edit distance): Calculates the minimum number of edits (insertion, deletion, or substitution) required to transform one string into another. Using this algorithm in the tMatchGroup component, you do not need to specify a maximum distance. The component automatically calculates a matching percentage based on the distance. This matching score will be used for the global matching calculation, based on the weight you assign in the Confidence Weight field.
* Metaphone: Based on a phonetic algorithm for indexing entries by their pronunciation. It first loads the phonetics of all entries of the lookup reference and checks all entries of the main flow against the entries of the reference flow. It does not support Chinese characters.
* Double Metaphone: A new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. It can return both a primary and a secondary code for a string. This accounts for some ambiguous cases as well as for multiple variants of surnames with common ancestry. It does not support Chinese characters.
* Soundex FR: Matches processed entries according to a standard French phonetic algorithm.It does not support Chinese characters.
* Jaro: Matches processed entries according to spelling deviations. It counts the number of matched characters between two strings. The higher the distance is, the more similar the strings are.
* Jaro-Winkler: A variant of Jaro, but it gives more importance to the beginning of the string.
* Fingerprint key: Matches entries after doing the following sequential process:
  1. Remove leading and trailing whitespace.
  2. Change all characters to their lowercase representation. It does not support Chinese characters.
  3. Remove all punctuation and control characters.
  4. Split the string into whitespace-separated tokens.
  5. Sort the tokens and remove duplicates.
  6. Join the tokens back together. Because the string parts are sorted, the given order of tokens does not matter. So, *Cruise, Tom* and *Tom Cruise* both end up with a fingerprint *cruise tom* and therefore end up in the same cluster.
  7. Normalize extended western characters to their ASCII representation, for example *gödel* to *godel*. This reproduces data entry mistakes performed when entering extended characters with an ASCII-only keyboard. However, this procedure can also lead to false positives, for example *gödel* and *godél* would both end up with *godel* as their fingerprint but they are likely to be different names. So this might work less effectively for datasets where extended characters play substantial differentiation role.
* q-grams: Matches processed entries by dividing strings into letter blocks of length q in order to create a number of q length grams. The matching result is given as the number of q-gram matches over possible q-grams.
* Hamming: Calculates the minimum number of substitutions required to transform one string into another string having the same length. For example, the Hamming distance between "masking" and "pairing" is 3.
* custom...: Lets you load an external matching algorithm from a Java library using the custom Matcher column.

For further information about how to load an external Java library, see [tLibraryLoad](https://help.qlik.com/talend/en-US/components/8.0/library-import/tlibraryload).

For further information about how to create a custom matching algorithm, see [Creating a custom matching algorithm](https://help.qlik.com/talend/en-US/data-matching/8.0/creating-custom-matching-algorithm).

For a related scenario about how to use a custom matching algorithm, see [Using a custom matching algorithm to match entries](https://help.qlik.com/talend/en-US/data-matching/8.0/using-custom-matching-algorithm-to-match-entries).