# tPatternUnmasking properties for Apache Spark Structured Streaming [Preview](https://help.qlik.com/en-US/cloud-services/Subsystems/Hub/Content/Sense_Hub/Introduction/feature-preview-program-overview.htm) These properties are used to configure tPatternUnmasking running in the Spark Structured Streaming Job framework. The Spark Structured Streaming tPatternUnmasking component belongs to the Data Quality family. This component is supported on Local Spark 3.5.x and Databricks/EMR with Spark 3.x. Information noteNote: This component does not support dynamic schema. This component is available in Talend Real-Time Big Data Platform and Talend Data Fabric. ## Basic settings | Properties | Description | | --- | --- | | Schema and
Edit Schema |

A schema is a row description. It defines the number of fields (columns) to be processed and passed on to the next component. When you create a Spark Job, avoid the reserved word line when naming the fields.

Click Sync columns to retrieve the schema from the previous component connected in the Job.

Click Edit schema to make changes to the schema. If the current schema is of the Repository type, three options are available:
- View schema: choose this option to view the schema only.
- Change to built-in property: choose this option to change the schema to Built-in for local changes.
- Update repository connection: choose this option to change the schema stored in the repository and decide whether to propagate the changes to all the Jobs upon completion.
  
  If you just want to propagate the changes to the current Job, you can select No upon completion and choose this schema metadata again in the Repository Content window.
The output schema of this component contains one read-only column, ORIGINAL_MARK. This column identifies by true or false if the record is an masked or and original respectively.
Built-In: You create and store the schema locally for this component only.
Repository: You have already created the schema and stored it in the Repository. You can reuse it in various projects and Job designs.

| | Modifications | Define in the table what fields to unmask and how to
unmask them:

Use the same settings for the Field
type, Values,
Path, Range and
Date Range columns as the ones used for
masking the input data with the tPatternMasking
component.

Column to unmask:
Select the column from the input flow that contains the data to be
unmasked.

Each column is processed sequentially, meaning that data
unmasking operations will be performed on the data from the first
column, the second column, and so on.

In a colum, each data field is a fixed length field,
except the last data field.

For fixed length fields, each value must contain the
same number of characters, for example: "30001,30002,30003" or "FR,EN".

In a column, the last Enumeration or Enumeration
from file data field is a variable length field.

For variable length fields, each value might not always
contain the same number of characters, for example: "30001,300023,30003" or "FR,ENG".

Field type: Select
the field type the data belongs to.

Interval: When selected, set a range of numeric values used for masking purposes in the Range field, using the following syntax: "<min>,<max>".

The number of unmasked characters from the input data corresponds to the number of characters of the maximum value.

For example, "1,999" will be interpreted as "001,999", which means that three characters from the input data will be masked by a value randomly selected from the defined range of values.
Enumeration: When selected, enter a comma-separated list of values to be used for masking data in the Values field, using the following syntax: "value1,value2,value3".
Enumeration from file: When selected, set the path to the CSV file containing a list of values used for masking data in the Path field. The file must contain one value per row and each value must be unique.
You can select a file on:
- The local system.
- Amazon S3, in local Spark mode or connected to EMR, using tS3Configuration. S3N and S3A file systems are supported.
- Azure Blob Storage, in local Spark mode or connected to Azure HDInsight or Databricks, using tAzureFSConfiguration.
- Azure Data Lake Storage, in local Spark mode or connected to Azure HDInsight, using tAzureFSConfiguration.
- Google Cloud Storage, in local Spark mode or connected to Dataproc, using tGSConfiguration.
- HDFS, in local Spark mode or connected to Azure HDInsight, using tHDFSConfiguration.
Set the file path as follows:
- In local mode:
  - Apache Spark 3.1 and earlier: prefix://file path or file:///file path.
  - Apache Spark 3.2 and later: file:///file path.
- In Standalone and Yarn modes: prefix://file path.
- If the file is on a cluster, hdfs://hdpnameservice1/file path.
- If the file is on Azure Blob Storage and you are connected to Azure HDInsight or Databricks, wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/<folder>/<file_name>" .
Date pattern (YYYYMMDD): When selected, set a range of years in the Date Range field, using the following syntax: "<min_year>,<max_year>".

Years can only have four digits, for example: "1900,2100".

The input dates to be masked must follow the YYYYMMDD pattern, for example: 20180101.

For example, if the input date is 20180101 and the value in the Date Range is "1900,2100", 19221221 could be the output date.

In the Values,
Path, Range and Date Range, values must be enclosed in double
quotes.

When the input data is invalid, meaning that a value
does not match the pattern defined in the component, the generated value is
null. | ## Usage | Usage guidance | Description | | --- | --- | | **Usage rule** | This component is used as an intermediate step.

This component, along with the Spark Structured Streaming component Palette it belongs to,
appears only when you are creating a Spark Structured Streaming Job. | | Spark Connection | You need to use the Structured Streaming Configuration
tab in the Run view to define the connection to a Spark cluster for
the whole Job.

This connection is effective on a per-Job basis. |