FILE: Existing source

Adding a new FLAT FILE entity to an existing source

Select radio button: To existing source

This will kick off the Add Data wizard to define a new entity within an existing source.

Select Source Name from dropdown. This is the source that the new entity will be added to.

Entity Level: Specify level of data management

(see System Settings: Level Control)

The default level for the source will populate in the dropdown and this can be modified and applied to new entities.

Base Directory: Pre-defined. This is the directory where data will be stored in file system. [This value is specified in core_env property: localfile.base.dir.source.connection. This property limits access and browsing to subdirectories of the specified location.]

Groups: Select Groups Select Group(s) requiring access from the dropdown options. At least one group must be added for the data to be discoverable.

Click Next.

File Format

Entity Name: Enter a new Entity Name. Note that the application will not allow two Entities with duplicate names under the same Source.

Entity File Path: Enter the exact file path to the originating source, ending with the file name and file format suffix. Select File Browser to explore directories for the appropriate source file. File browser functionality is available for data file selection for all file-based sources except Kafka and OpenConnector sources.

Data Format Specification*: Choose a mechanism to Generate Record Layout. When Source or Sample Files are attached, appropriate metadata is extracted and applied to the environment. These files are uploaded from the user's local machine.

• FDL: File Definition Language specification document

• SAMPLE FILE: This file can provide information about Header/Trailer, Field (Column) Names, Data Type, delimiters, terminators, business date, etc. (essentially the same information that is passed via FDL documents)

• SOURCE FILE: The Source file (path provided in previous step) is read to define structural entity, record, and field metadata.

  [No file is provided; the source is being used to generate the layout]

• NONE: This is an option if an analyzer file is not available. Structural elements can be manually defined in following steps.

Generate Record Layout to configure the following specifications.

Users are able to overwrite format specifications.

*Note source file limitations:

Parquet and AVRO files: Only the Source option works for metadata detection for Parquet and ORC files. (The Sample option is only supported for text files.)

ORC files: Metadata detection data loading is not supported from ORC source files (though ORC target file storage format is supported).

Local File Browser for discovery of Local Server directories (Select Folder icon)

File Information

In most cases, users will have uploaded a file in the previous step. that will be parsed and applied to the following layout specifications.

Character Encoding: The most common record layout is UTF_8.

Options include:

• US_ASCII
• EBCDIC_037
• LATIN_1
• WINDOWS_1252
• UTF_8
• UTF_16LE
• UTF_16BE
• UTF_32LE
• UTF_32BE
• UTF_8_OR_LATIN_1

Record Layout

The most common record layout is VARIABLE_CHAR_LENGTH_TERMINATED.

Options include:

• FIXED_BYTE_LENGTH
• FIXED_BYTE_LENGTH_TERMINATED
• VARIABLE_CHAR_LENGTH_TERMINATED
• MAINFRAME_VARIABLE_BDW_RDW
• MAINFRAME_VARIABLE_RDW_ONLY
• PARQUET
• AVRO

Header Information

Byte Count or Line Count: Default is Null (empty) though many tables have a header (Line Count=1)

Record Information

Record information format options:

• Field Delimiter*

  • Tab (Default)

    (\t)

  • Comma

    ,

  • Pipe

    |

  • Semi-colon

    ;

  • Colon

    :

  • Ctrl+A

    (\x01)

  • Space

    (\x20)

  • Double Pipe

    ||

  • Pipe Tilde

    |~

  *Any character(s) can be used as the delimiter in data. The delimiter can be coded however the incoming file character set encodes that character.

  For example, if the delimiter is the broken pipe character: ¦, the delimiter can be encoded for the incoming character set:

  UTF-8 (hex): 0xC2 0xA6 (c2a6)

  UTF-16 (hex): 0x00A6 (00a6)

  Latin-1: \xA6

  Unicode: U+00A6

• Record Terminator:

  • Newline

    \n

  • Carriage Return (optionally preceding)

    /Newline

    \r?\n

  • Return/Newline

    \r\n

  • Any Newline

    ANY_NEWLINE

    (default on ingest)

• Field Open Enclosure:

  • Single Quote

    '

  • Double Quote

    "

  • Angular Brackets

    >>

• Record Validation Regex: Standard Regex arguments

• Quote Scheme:

  • NONE
  • DOUBLE-EMBEDDED ENCLOSURE
• Minimum Byte Count: Default=1
• Maximum Byte Count: Default=65536

Trailer Information

Default is null

Internal file format Type

Options include:

• AVRO
• ORC
• ORC_ALL_STRING
• PARQUET
• PARQUET_ALL_STRING
• TEXT_TAB_DELIMITED

Qlik Catalog Stored Format Type step is skipped for single node environments where Stored Format Type is always TEXT_TAB_DELIMITED.

Record Layout Preview

Select Record Layout Preview to review and edit field columns.

Save the Entity

The following fields will populate and can be edited (column order will vary based on user configuration):

• Name (required)
• Index (user-defined if manually entered or auto-populated from source doc)
• Business Description (optional)
• Technical Description (optional)
• Data Type (required)
• Required: NOT NULL where a NULL value will be marked as UGLY (Default is 'false')
• Encrypted at Source (Default is 'false')
• Key: Primary Key (Default is 'false')
• Validation Regex (Default is' ' [empty])
• Foreign key (Default is 'false')

Load Data to the Entity.

Once the Entity has been saved, navigate to Entities screen in Source to load the data.

Select Load from the More dropdown.