tPatternMasking properties for Apache Spark Structured Streaming

Last updated: 9/30/2026

Feature in preview

These properties are used to configure tPatternMasking running in the Spark Structured Streaming Job framework.

The Spark Structured Streaming tPatternMasking component belongs to the Data Quality family.

This component is supported on Local Spark 3.5.x and Databricks/EMR with Spark 3.x.

This component is available in Talend Real-Time Big Data Platform and Talend Data Fabric.

Basic settings

Properties	Description
Schema and Edit Schema	A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word `line` when naming the fields. Click Sync columns to retrieve the schema from the previous component connected in the Job. Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available: View schema: choose this option to view the schema only. Change to built-in property: choose this option to change the schema to Built-in for local changes. Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion. If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window. The output schema of this component contains read-only columns: TWEAK: Is generated when the Use tweaks with FF1 Encryption check box is selected. This column contains the tweak necessary to decrypt the data. ORIGINAL_MARK: Identifies by true or false if the record is an original record or a substitute record respectively. Built-In: You create and store the schema locally for this component only. Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.
Modifications	Define in the table what fields to change and how to change them: Column to mask: Select the column from the input flow for which you want to generate similar data by modifying its values. You can mask data from different columns but you need to follow the order of the fields you want to mask. Each column is processed sequentially, meaning that data masking operations will be performed on the data from the first column, the second column, and so on. In a colum, each data field is a fixed length field, except the last data field. For fixed length fields, each value must contain the same number of characters, for example: `"30001,30002,30003"` or `"FR,EN"`. In a column, the last Enumeration or Enumeration from file data field is a variable length field. For variable length fields, each value might not always contain the same number of characters, for example: `"30001,300023,30003"` or `"FR,ENG"`. Field type: Select the field type the data belongs to. Interval: When selected, set a range of numeric values used for masking purposes in the Range field, using the following syntax: `"<min>,<max>"`. The number of masked characters from the input data corresponds to the number of characters of the maximum value. For example, `"1,999"` will be interpreted as `"001,999"`, which means that three characters from the input data will be masked by a value randomly selected from the defined range of values. Enumeration: When selected, enter a comma-separated list of values to be used for masking data in the Values field, using the following syntax: `"value1,value2,value3"`. Each value must contain the same number of characters. For example: `"30001,30002,30003"` or `"FR,EN"`. Enumeration from file: When selected, set the path to the CSV file containing a list of values to be used for masking data in the Path field. The file must contain one value per row, each value must be unique and the values must have the same number of characters. You can select a file on: The local system. Amazon S3, in local Spark mode or connected to EMR, using tS3Configuration. S3N and S3A file systems are supported. Azure Blob Storage, in local Spark mode or connected to Azure HDInsight or Databricks, using tAzureFSConfiguration. Azure Data Lake Storage, in local Spark mode or connected to Azure HDInsight, using tAzureFSConfiguration. Google Cloud Storage, in local Spark mode or connected to Dataproc, using tGSConfiguration. HDFS, in local Spark mode or connected to Azure HDInsight, using tHDFSConfiguration. Set the file path as follows: In local mode: Apache Spark 3.1 and earlier: prefix://file path or file:///file path. Apache Spark 3.2 and later: file:///file path. In Standalone and Yarn modes: prefix://file path. If the file is on a cluster, hdfs://hdpnameservice1/file path. If the file is on Azure Blob Storage and you are connected to Azure HDInsight or Databricks, wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<folder>/<file_name>" . Paths to folders are not supported. Date pattern (YYYYMMDD): When selected, set a range of years in the Date Range field, using the following syntax: `"<min_year>,<max_year>"`. Years can only have four digits, for example: `"1900,2100"`. The input dates to be masked must follow the YYYYMMDD pattern, for example: `20180101`. For example, if the input date is `20180101` and the value in the Date Range is `"1900,2100"`, 19221221 could be the output date. In the Values, Path, Range and Date Range, values must be enclosed in double quotes. When the input data is invalid, meaning that a value does not match the pattern defined in the component, the generated value is null.

Properties

Description

Schema and Edit Schema

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Click Sync columns to retrieve the schema from the previous component connected in the Job.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:
- View schema: choose this option to view the schema only.
- Change to built-in property: choose this option to change the schema to Built-in for local changes.
- Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.
  
  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
The output schema of this component contains read-only columns:
- TWEAK: Is generated when the Use tweaks with FF1 Encryption check box is selected. This column contains the tweak necessary to decrypt the data.
- ORIGINAL_MARK: Identifies by true or false if the record is an original record or a substitute record respectively.
Built-In: You create and store the schema locally for this component only.
Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

Modifications

Define in the table what fields to change and how to change them:

Column to mask: Select the column from the input flow for which you want to generate similar data by modifying its values.

You can mask data from different columns but you need to follow the order of the fields you want to mask.

Each column is processed sequentially, meaning that data masking operations will be performed on the data from the first column, the second column, and so on.

In a colum, each data field is a fixed length field, except the last data field.

For fixed length fields, each value must contain the same number of characters, for example: "30001,30002,30003" or "FR,EN".

In a column, the last Enumeration or Enumeration from file data field is a variable length field.

For variable length fields, each value might not always contain the same number of characters, for example: "30001,300023,30003" or "FR,ENG".

Field type: Select the field type the data belongs to.

Interval: When selected, set a range of numeric values used for masking purposes in the Range field, using the following syntax: "<min>,<max>".

The number of masked characters from the input data corresponds to the number of characters of the maximum value.

For example, "1,999" will be interpreted as "001,999", which means that three characters from the input data will be masked by a value randomly selected from the defined range of values.
Enumeration: When selected, enter a comma-separated list of values to be used for masking data in the Values field, using the following syntax: "value1,value2,value3".

Each value must contain the same number of characters. For example: "30001,30002,30003" or "FR,EN".
Enumeration from file: When selected, set the path to the CSV file containing a list of values to be used for masking data in the Path field. The file must contain one value per row, each value must be unique and the values must have the same number of characters.
You can select a file on:
- The local system.
- Amazon S3, in local Spark mode or connected to EMR, using tS3Configuration. S3N and S3A file systems are supported.
- Azure Blob Storage, in local Spark mode or connected to Azure HDInsight or Databricks, using tAzureFSConfiguration.
- Azure Data Lake Storage, in local Spark mode or connected to Azure HDInsight, using tAzureFSConfiguration.
- Google Cloud Storage, in local Spark mode or connected to Dataproc, using tGSConfiguration.
- HDFS, in local Spark mode or connected to Azure HDInsight, using tHDFSConfiguration.
Set the file path as follows:
- In local mode:
  - Apache Spark 3.1 and earlier: prefix://file path or file:///file path.
  - Apache Spark 3.2 and later: file:///file path.
- In Standalone and Yarn modes: prefix://file path.
- If the file is on a cluster, hdfs://hdpnameservice1/file path.
- If the file is on Azure Blob Storage and you are connected to Azure HDInsight or Databricks, wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<folder>/<file_name>" .
Paths to folders are not supported.
Date pattern (YYYYMMDD): When selected, set a range of years in the Date Range field, using the following syntax: "<min_year>,<max_year>".

Years can only have four digits, for example: "1900,2100".

The input dates to be masked must follow the YYYYMMDD pattern, for example: 20180101.

For example, if the input date is 20180101 and the value in the Date Range is "1900,2100", 19221221 could be the output date.

In the Values, Path, Range and Date Range, values must be enclosed in double quotes.

When the input data is invalid, meaning that a value does not match the pattern defined in the component, the generated value is null.

Usage

Usage guidance	Description
Usage rule	This component is used as an intermediate step. This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.
Spark Connection	You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job. This connection is effective on a per-Job basis.

Usage guidance

Description

Usage rule

This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

Spark Connection

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here