Metadata file description
When the Create metadata files in the target folder option in the File target endpoint or Amazon S3 target endpoint is selected, for each CSV/JSON/Parquet file Replicate creates a corresponding metadata file under the specified target folder.
The metadata file offers several benefits such as enabling custom batch processes to perform better validation, supporting deeper automation, offering lineage information and improving processing reliability.
The metadata files are described in the tables below.
All timestamps are in ISO-8601 format, for example 2016-08-02T10:05:04.802.
| Field | Description |
|---|---|
|
name |
The name of the Replicate task. |
|
sourceEndpoint |
The name defined in the source endpoint settings. |
|
sourceEndpointType |
The endpoint type defined in the source endpoint settings (e.g. Oracle, MySQL, etc.). |
|
sourceEndpointUser |
The user defined in the source endpoint settings. |
|
replicationServer |
The hostname of the machine on which Qlik Replicate is installed. |
|
operation |
If a target data file has been created, this field will contain the following value: dataProduced |
| Field | Description |
|---|---|
|
name |
The name of the data file without the extension. |
|
extension |
The extension of the data file (.csv or.json according to the selected target file format). |
|
location |
The location of the data file. |
|
startWriteTimestamp |
UTC timestamp indicating when writing to the file started. |
|
endWriteTimestamp |
UTC timestamp indicating when writing to the file ended. |
|
firstTransactionTimestamp |
UTC timestamp of the first record in the file. |
|
lastTransactionTimestamp |
UTC timestamp of the last record in the file. |
|
content |
The values can either be data (i.e.Full Load replication) or changes (i.e Change Processing replication) according to the data in the corresponding CSV file. |
|
recordCount |
The number of records in the file. |
|
errorCount |
The number of data errors encountered during file creation. |
| Field | Description |
|---|---|
|
format |
delimited or json according to the selected target file format. |
|
options |
The options for delimited file format. These options will not be shown for json format as they are not relevant. |
|
recordDelimiter |
The delimiter used to separate records (rows) in the target files. The default is newline (\n). |
|
fieldDelimiter |
The delimiter used to separate fields (columns) in the target files. The default is a comma. |
|
nullValue |
The string used to indicate a null value in the target file. |
|
quoteChar |
The character used at the beginning and end of a column. The default is the double-quote character ("). |
|
escapeChar |
The character used to escape a string when both the string and the column containing the string are enclosed in double quotes. Note that the string’s quotation marks will be removed unless they are escaped. Example (where " is the quote character and \ is the escape character): 1955,"old, \"rare\", Chevrolet",$1000 |
| Field | Description |
|---|---|
|
customInfo |
This section contains any custom parameters that were set using the dfmCustomProperties internal parameter. The dfmCustomProperties internal parameter must be specified in the following format: Parameter1=Value1;Parameter2=Value2;Parameter3=Value3 Example: Color=Blue;Size=Large;Season=Spring For an explanation of how to set internal parameters, see Internal parameters. |
| Field | Description |
|---|---|
|
sourceSchema |
The schema containing the source table. |
|
sourceTable |
The name of the source table. |
|
targetSchema |
The name of the target table schema (if the source schema name was changed). For information on changing the source schema name, see Performing General tasks for a single table/view. |
|
targetTable |
The name of the target table (if the source table name was changed). For information on changing the source table name, see Performing General tasks for a single table/view. |
|
tableVersion |
Replicate assigns an internal version number to the table. The version number increases whenever a DDL change occurs in the source table. |
|
columns |
Information about the table columns. |
|
ordinal |
The position of the column in the record (1, 2, 3, etc.). |
|
name |
The column name. |
|
type |
The column data type. See File target data typesor Amazon S3 Target Data Types for more information. |
|
width |
The maximum size of the data (in bytes) permitted for the column. |
|
scale |
The maximum number of digits to the right of the decimal point permitted for a number. |
|
primaryKeyPos |
The position of the column in the table’s Primary Key or Unique Index. The value is zero if the column is not part of the table’s Primary Key. |