Data Quality CSV Format

• Rules (folder with Data Quality information)
  • Folder.csv (allows to recreate an hierarchy of folders in order to organize DQ rules)
    • Id (folder unique identifier)
    • Name (folder name)
    • Parent Folder Id (identifier of the parent folder, optional)
    • Description (folder description, optional)
  • Rule.csv (provides DQ rule declarations)
    • Id (rule unique identifier)
    • Name (rule name. Name must be unique under the parent folder.)
    • Folder Id (identifier of the parent folder)
    • Description (rule description, optional)
    • Operation (rule operation, optional)
    • Dimension (rule dimension name. See the list of the possible values below)
    • Url (hyperlink on the rule in the original DQ tool, optional)
  • Rule Application.csv (allows to specify DQ rule instances and associate them with the individual columns)
    • Id (rule application unique identifier)
    • Rule Id (identifier of the associated DQ rule)
    • Type (column path, connection type. Is needed to interpret and then correctly stitch the referenced column in MM. See the list of the possible values below.)
    • Server (column path, server name. Database host:port, Snowflake warehouse, file service, like gs://bucket)
    • Catalog (column path, catalog name. It can be a Snowflake or SQL Server database. Empty for Oracle and File systems.)
    • Schema (column path, schema name. It can be a database schema, file folder path, like /dir1/dir2 )
    • Table (column path, table name. It can also be a file, or partitioned directory name)
    • Column (column path, column name)
    • Description (rule application description, optional)
    • Threshold (rule application threshold. It is a floating point value that is used to compute DQ Status in MM (Good, Acceptable or Bad). Use comma as a delimiter if more than one value is required)
  • Rule Application Column.csv (provides additional associations between DQ Rule Applications and the corresponding columns if Rule Application refers to two or more columns)
    • Rule Application Id (rule application identifier)
    • Type (column path, connection type. Is needed to interpret and then correctly stitch the referenced column in MM. See the list of the possible values below.)
    • Server (column path, server name. Database host:port, Snowflake warehouse, file service, like gs://bucket)
    • Catalog (column path, catalog name. It can be a Snowflake or SQL Server database. Empty for Oracle and File systems.)
    • Schema (column path, schema name. It can be a database schema, file folder path, like /dir1/dir2 )
    • Table (column path, table name. It can also be a file, or partitioned directory name)
    • Column (column path, column name)
  • Rule Measure.csv (provides rule execution results)
    • Id (rule measure unique identifier)
    • Rule Application Id (rule application identifier)
    • Date (the execution date when the values have been collected. See below the supported date time format)
    • Rows Total (the overall number of the processed rows)
    • Rows Failed (the number of rows that failed the DQ rule)
• Profiling (folder with Data Profiling information, optional)
  • Table.csv (table level profiling information)
    • Type (table path, connection type. Is needed to interpret and then correctly stitch the referenced table in MM. See the list of the possible values below.)
    • Server (table path, server name. Database host:port, Snowflake warehouse, file service, like gs://bucket)
    • Catalog (table path, catalog name. It can be a Snowflake or SQL Server database. Empty for Oracle and File systems.)
    • Schema (table path, schema name. It can be a database schema, file folder path, like /dir1/dir2 )
    • Table (table path, table name. It can also be a file, or partitioned directory name)
    • Creation Date (the date/time when data profiling information has been collected. See below the supported date time format)
    • Data Update Date (the last date/time when data has been updated in the source table. See below the supported date time format)
    • Records All (the number of records in the source table)
    • Records Processed (the number of records that have been processed by the profiler)
  • Column.csv (column level profiling information)
    • Type (column path, connection type. Is needed to interpret and then correctly stitch the referenced column in MM. See the list of the possible values below.)
    • Server (column path, server name. Database host:port, Snowflake warehouse, file service, like gs://bucket)
    • Catalog (column path, catalog name. It can be a Snowflake or SQL Server database. Empty for Oracle and File systems.)
    • Schema (column path, schema name. It can be a database schema, file folder path, like /dir1/dir2 )
    • Table (column path, table name. It can also be a file, or partitioned directory name)
    • Column (column path, column name)
    • Creation Date (the date/time when data profiling information has been collected. See below the supported date time format)
    • Count (KPI with the count of all the values in the column. Long value)
    • Distinct (KPI with the counter for distinct values. Long value)
    • Duplicate (KPI with the counter for duplicate values. Long value)
    • Blank (KPI with the counter for blank string values. Long value)
    • Zero (KPI with the counter for zero numeric values. Long value)
    • Null (KPI with the counter for NULL values. Long value)
    • Min (KPI with the minimum value in the column. Usually applicable for the numeric values.)
    • Max (KPI with the maximum value in the column. Usually applicable for the numeric values.)
    • Mean (KPI with the mean value in the column. Usually applicable for the numeric values.)
    • Variance (KPI with the variance value in the column. Usually applicable for the numeric values.)
    • Median (KPI with the median value in the column. Usually applicable for the numeric values.)
    • Lower quantile (KPI with the lower quantile value in the column. Usually applicable for the numeric values.)
    • Upper quantile (KPI with the upper quantile value in the column. Usually applicable for the numeric values.)
    • Avg length (KPI with the average value length in the column. Usually applicable for the string values. Double value)
    • Min length (KPI with the minimum value length in the column. Usually applicable for the string values. Long value)
    • Max length (KPI with the maximum value length in the column. Usually applicable for the string values. Long value)
    • Valid (KPI with the counter for valid values. Usually computed based on the column’s data type. Long value)
    • Invalid (KPI with the counter for invalid values. Usually computed based on the column’s data type. Long value)
    • Data Type Invalid Values (Serialized list of the invalid values. Usually computed based on the column’s data type. Use character semicolon ‘;’ as a delimiter and backward slash ‘\’ as an escape character to serialize multiple values)
    • Data Type Frequency (Serialized list of data type frequencies in the format “[value],[count],[ratio]” (e.g. INTEGER,6,60.00;STRING,4,40.00). Use character semicolon ‘;’ as a delimiter and backward slash ‘\’ as an escape character to serialize multiple values)
    • Pattern Frequency (Serialized list of pattern frequencies in the format “[value],[count],[ratio]” (e.g. 999-99-9999,19972,1.00000). Use character semicolon ‘;’ as a delimiter and backward slash ‘\’ as an escape character to serialize multiple values)
    • Value Frequency (Serialized list of value frequencies in the format “[value],[count],[ratio]” (e.g. 987-66-3365,1,0.00005;987-66-5400,1,0.00005). Use character semicolon ‘;’ as a delimiter and backward slash ‘\’ as an escape character to serialize multiple values)
• Sampling (folder with Data Sampling information, optional)
  • Data Sampling.csv (mapping file that allows to lookup Data Sampling information for the individual tables/files. Is used to resolve special characters that can be part of the table names but not allowed in the file system to name the files accordingly)
  • *[file path recreated from ‘Data Sampling.csv’] (e.g. "\JDBC\172.0.0.1\FinanceDWDimensional\dbo\Customer.csv")
  • Table Url.csv (file that allows to specify a hyperlink to the original DQ source for the specified table, optional)
    • Type (table path, connection type. Is needed to interpret and then correctly stitch the referenced table in MM. See the list of the possible values below.)
    • Server (table path, server name. Database host:port, Snowflake warehouse, file service, like gs://bucket)
    • Catalog (table path, catalog name. It can be a Snowflake or SQL Server database. Empty for Oracle and File systems.)
    • Schema (table path, schema name. It can be a database schema, file folder path, like /dir1/dir2 )
    • Table (table path, table name. It can also be a file, or partitioned directory name)
    • Url (a hyperlink on the corresponding table record in the original DQ tool)