AI processor
Uses generative AI functions from your third-party target data platform to transform the data.
The AI processor uses native generative AI model capabilities in your third-party target data platform. You can for example translate, classify, or summarize data in your transformation.
Availability
AI processor capability is currently only available for projects with Databricks as target data platform, using Databricks Foundation Model APIs. This Databricks capability is in Public Preview and can contain limitations, for example it is not supported on Databricks SQL Classic. For information about function specific limitations, see the link to Databricks documentation for each respective function.
See also Databricks Previews support & details.
The following functions are available:
-
Analyze sentiment
-
Classify
-
Fix grammar
-
Mask
-
Similarity
-
Summarize
-
Translate
AI properties
Select which AI function to use in Function name.
Analyze sentiment
Perform sentiment analysis on input text.
The sentiment is returned as a text string with the value of positive, negative, neutral, or mixed. If the sentiment cannot be detected, null is returned.
For more information, see ai_analyze_sentiment function.
Property name | Configuration |
---|---|
Content |
Select the column you want to perform sentiment analysis on. You can only select columns of string type. |
Output column name |
Enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Classify
Classify input text according to labels you provide.
For more information, see ai_classify function.
Property name | Configuration |
---|---|
Content |
Select the column you want to classify text for. You can only select columns of string type. |
Classification labels |
Add labels to use when classifying the data. Use to add more labels. You can use from 2 to 20 labels. |
Fix grammar
Correct grammatical errors in a text column.
For more information, see ai_fix_grammar function.
Property name | Configuration |
---|---|
Content |
Select the column you want to fix grammar in. You can only select columns of string type. |
Output column name |
If you select Create a new column, you can enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Mask
Mask specified entities in a text column. Masked entities are replaced with [MASKED].
For more information, see ai_mask function.
Property name | Configuration |
---|---|
Content |
Select the column you want to mask text entities in. You can only select columns of string type. |
Mask labels | Add a label for each text entity that you want to mask. Use to add more labels. |
Output column name |
If you select Create a new column, you can enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Similarity
Compare two strings and computes the semantic similarity score. The score is returned as a float value between 0 and 1.0, where 1.0 means that the strings are equal.
For more information, see ai_similarity function.
Property name | Configuration |
---|---|
Content |
Select the column you want to compare. You can only select columns of string type. |
With |
You can compare the text in Content with text from another string column or a value that you specify.
|
Output column name |
Enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Summarize
Generate a summary of the text in a text column.
For more information, see ai_summarize function.
Property name | Configuration |
---|---|
Content |
Select the column you want to summarize. You can only select columns of string type. |
Max word count |
Set the maximum word count of the text summary. You can only set integer values. The default value is 50. If you leave it empty or set it to zero, the maximum word count is not applied. |
Output column name |
Enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Translate
Translates the text content of a column.
For more information, see ai translate function.
Property name | Configuration |
---|---|
Content |
Select the column you want to summarize. You can only select columns of string type. |
Translate to |
Select the language to translate to. |
Output column name |
Enter a name for the generated output column. The expected format is the following:
Example: ASDasd123_4564 |
Editing the processor
To rename the processor, click the Edit icon that is displayed when hovering over the default name of the processor.
To edit its description, click the Edit icon that is displayed when hovering over Description.