tGenKey properties for Apache Spark Streaming

Schema and Edit schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Click Sync columns to retrieve the schema from the previous component connected in the Job.

Built-In: You create and store the schema locally for this component only.

Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Click the import icon to import blocking keys from the match rules that are defined and saved in the Studio repository.

When you click the import icon, a Match Rule Selector wizard is opened to help you import blocking keys from the match rules listed in the Studio repository and use them in your Job.

You can import blocking keys only from match rules that are defined with the VSR algorithm and saved in the Studio repository. For further information, see Importing match rules from the Studio repository

Column

Select the column(s) from the main flow on which you want to define certain algorithms to set the functional key.

Pre-Algorithm

If required, select the relevant matching algorithm from the list:

Remove diacritical marks: removes any diacritical mark.

Remove diacritical marks and lower case: removes any diacritical mark and converts to lower case before generating the code of the column.

Remove diacritical marks and upper case: removes any diacritical mark and converts to upper case before generating the code of the column.

Lower case: converts the field to lower case before applying the key algorithm.

Upper case: converts the field to upper case before applying the key algorithm.

Add left position character: enables you to add a character to the left of the column.

Add right position character: enables you to add a character to the right of the column.

Value

Set the algorithm value, where applicable.

Algorithm

Select the relevant algorithm from the list:

First character of each word: includes in the functional key the first character of each word in the column. When using Japanese characters in the input data, the input text should be tokenized. For more information, see tJapaneseTokenize.

N first characters of each word: includes in the functional key N first characters of each word in the column. When using Japanese characters in the input data, the input text should be tokenized. For more information, see tJapaneseTokenize.

First N characters of the string: includes in the functional key N first characters of the string.

Last N characters of the string: includes in the functional key N last characters of the string.

First N consonants of the string: includes in the functional key N first consonants of the string. Japanese and Chinese characters are not supported.

First N vowels of the string: includes in the functional key N first vowels of the string. Japanese and Chinese characters are not supported.

Pick characters: includes in the functional key the characters located at a fixed position (corresponding to the set digital/range).

Exact: includes in the functional key the full string.

Substring(a,b): includes in the functional key character according to the set index.

Soundex code: generates a code according to a standard English phonetic algorithm. This code represents the character string that will be included in the functional key. Japanese and Chinese characters are not supported.

Metaphone code: generates a code according to the character pronunciation. This code represents the character string that will be included in the functional key. Japanese and Chinese characters are not supported.

Double-metaphone code: generates a code according to the character pronunciation using a new version of the Metaphone phonetic algorithm, that produces more accurate results than the original algorithm. This code represents the character string that will be included in the functional key. Japanese and Chinese characters are not supported.

Cologne phonetics: a Soundex phonetic algorithm optimized for the German language. It encodes a string into a Cologne phonetic value. This code represents the character string that will be included in the functional key. Japanese and Chinese characters are not supported.

Value

Set the algorithm value, where applicable.

If you do not set a value for the algorithms which need one, the Job runs with a compilation error.

Post-Algorithm

If required, select the relevant matching algorithm from the list:

Use default value (string): enables you to choose a string to replace null or empty data.

Add left position character: enables you to add a character to the left of the column.

Add right position character: enables you to add a character to the right of the column.

Value

Set the option value, where applicable.

Show help

Select this check box to display instructions on how to set algorithms/options parameters.

Global Variables

ERROR_MESSAGE: the error message generated by the component when an error occurs. This is an After variable and it returns a string. This variable functions only if the Die on error check box is cleared, if the component has this check box.

A Flow variable functions during the execution of a component while an After variable functions after the execution of the component.

To fill up a field or expression with a variable, press Ctrl + Space to access the variable list and choose the variable to use from it.

For further information about variables, see Talend Studio User Guide.

Usage rule

This component, along with the Spark Streaming component Palette it belongs to, appears only when you are creating a Spark Streaming Job.

This component is used as an intermediate step.

You need to use the Spark Configuration tab in the Run view to define the connection to a given Spark cluster for the whole Job.

This connection is effective on a per-Job basis.

For further information about a Talend Spark Streaming Job, see the sections describing how to create, convert and configure a Talend Spark Streaming Job of the Talend Big Data Getting Started Guide .

Note that in this documentation, unless otherwise explicitly stated, a scenario presents only Standard Jobs, that is to say traditional Talend data integration Jobs.

Spark Connection

This connection is effective on a per-Job basis.