tDataEncrypt properties for Apache Spark Structured Streaming

Schema and Edit Schema

• A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

  Click Sync columns to retrieve the schema from the previous component connected in the Job.

  Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:

  • View schema: choose this option to view the schema only.

  • Change to built-in property: choose this option to change the schema to Built-in for local changes.

  • Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.

    If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.

• Built-In: You create and store the schema locally for this component only.

• Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Password

Available when Cryptographic file is selected as secret method.

This value must be enclosed in double quotes.

When using an existing cryptographic file, enter the password required to use this file.

When generating a cryptographic file, enter the password used to encrypt this file.

This password is required to decrypt back data using the tDataDecrypt component.

Cryptographic file path

Available when Cryptographic file is selected as secret method.

When using an existing cryptographic file, enter the path to the cryptographic file. It must be enclosed in double quotes.

This cryptographic file is encrypted using AES-GCM.

This cryptographic file is required to decrypt back data using the tDataDecrypt component.

For more information about the cryptographic file, see the data encryption process.

Generate cryptographic file

Available when Cryptographic file is selected as secret method.

Click this button to generate the cryptographic file.

Available when 256-bit key (encoded with base64) is selected as secret method.

This value must be enclosed in double quotes.

This key is required to decrypt back data using the tDataDecrypt component.

Available when 256-bit key (encoded with base64) is selected as secret method.

Columns to encrypt

Select the corresponding Encrypt check boxes to encrypt input columns.

You can encrypt all column data types, except Dynamic, but the output encrypted data is of String type.

Configure the output schema of the component to set the type of the columns to be encrypted to String.When the input data type is Dynamic or String, the output column is automatically set to Dynamic or String respectively.

The columns that are not selected will not be encrypted and be returned as-is by the component.

This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.