Exploring semantic categories of data columns

The semantic wizard opens listing all the columns of the table or listing the selected set of columns depending on whether you started the analysis on a table or on a set of columns respectively. The Category line in the wizard assigns semantic categories for the matched columns.

• Sampling Strategy: define what to list in the data preview. Select First N Rows to list the N first data records or select Reservoir Sampling to list N random records. Then set the number of records in the Number of rows field.
• Threshold for category discovery: decide the minimum threshold for the matches to show in the Category lists of the analyzed columns.

  This threshold filters the less probable categories of the analyzed columns.

• Refresh: refresh the data preview after any change in the configuration.

The names entered by you will display in a different color. This step stores locally the categories and the semantic names of the columns. If no semantic names are found, categories are stored anyway.

This is not mandatory but will help you better match table metadata with the concepts stored in the ontology repository on the Elasticsearch server.

The percentages of the proposed categories are calculated by analyzing the data in the columns against the following methods: regex, data dictionary and keyword dictionary. The dictionary indexes and regex categories are embedded in Talend Studio and are used to decide what category does the data fall in.