Skip to main content Skip to complementary content

FILE: Existing source

Adding a new FLAT FILE entity to an existing source

Select radio button: To existing source

This will kick off the Add Data wizard to define a new entity within an existing source.

Source wizard select add data to existing source

Select Source Name from dropdown. This is the source that the new entity will be added to.

Entity Level: Specify level of data management

(see System Settings: Level Control)

The default level for the source will populate in the dropdown and this can be modified and applied to new entities.

Base Directory: Pre-defined. This is the directory where data will be stored in file system. [This value is specified in core_env property: localfile.base.dir.source.connection. This property limits access and browsing to subdirectories of the specified location.]

Groups: Select Groups Select Group(s) requiring access from the dropdown options. At least one group must be added for the data to be discoverable.

Click Next.

Complete source configuration fields

File Format

Entity Name: Enter a new Entity Name. Note that the application will not allow two Entities with duplicate names under the same Source.

Entity File Path: Enter the exact file path to the originating source, ending with the file name and file format suffix. Select File Browser to explore directories for the appropriate source file. File browser functionality is available for data file selection for all file-based sources except Kafka and OpenConnector sources.

Data Format Specification*: Choose a mechanism to Generate Record Layout. When Source or Sample Files are attached, appropriate metadata is extracted and applied to the environment. These files are uploaded from the user's local machine.

  • FDL: File Definition Language specification document

  • SAMPLE FILE: This file can provide information about Header/Trailer, Field (Column) Names, Data Type, delimiters, terminators, business date, etc. (essentially the same information that is passed via FDL documents)

  • SOURCE FILE: The Source file (path provided in previous step) is read to define structural entity, record, and field metadata.

    [No file is provided; the source is being used to generate the layout]

  • NONE: This is an option if an analyzer file is not available. Structural elements can be manually defined in following steps.

Generate Record Layout to configure the following specifications.

Users are able to overwrite format specifications.

*Note source file limitations:

Parquet and AVRO files: Only the Source option works for metadata detection for Parquet and ORC files. (The Sample option is only supported for text files.)

ORC files: Metadata detection data loading is not supported from ORC source files (though ORC target file storage format is supported).

Entity ingest name, filepath, metadata method

Local File Browser for discovery of Local Server directories (Select Folder icon)

Browse directories for data file

File Information

In most cases, users will have uploaded a file in the previous step. that will be parsed and applied to the following layout specifications.

Character Encoding: The most common record layout is UTF_8.

Options include:

  • EBCDIC_037
  • LATIN_1
  • WINDOWS_1252
  • UTF_8
  • UTF_16LE
  • UTF_16BE
  • UTF_32LE
  • UTF_32BE
  • UTF_8_OR_LATIN_1

Record Layout

The most common record layout is VARIABLE_CHAR_LENGTH_TERMINATED.

Options include:

  • AVRO

Specify file information

Header Information

Byte Count or Line Count: Default is Null (empty) though many tables have a header (Line Count=1)

Specify header information

Record Information

Record information format options:

  • Field Delimiter*

    • Tab (Default)


    • Comma


    • Pipe


    • Semi-colon


    • Colon


    • Ctrl+A


    • Space


    • Double Pipe


    • Pipe Tilde


    *Any character(s) can be used as the delimiter in data. The delimiter can be coded however the incoming file character set encodes that character.

    For example, if the delimiter is the broken pipe character: ¦, the delimiter can be encoded for the incoming character set:

    UTF-8 (hex): 0xC2 0xA6 (c2a6)

    UTF-16 (hex): 0x00A6 (00a6)

    Latin-1: \xA6

    Unicode: U+00A6

  • Record Terminator:

    • Newline


    • Carriage Return (optionally preceding)



    • Return/Newline


    • Any Newline


      (default on ingest)

  • Field Open Enclosure:

    • Single Quote


    • Double Quote


    • Angular Brackets


  • Record Validation Regex: Standard Regex arguments
  • Quote Scheme:

    • NONE
  • Minimum Byte Count: Default=1
  • Maximum Byte Count: Default=65536
Tip note

'Field Open Enclosure', 'Field Close Enclosure', or 'Quote Scheme' specify that the content within the enclosure is part of a string so that other characters (like commas) within the enclosures are not processed as delimiters. This can be an important piece of metadata. If records return as 'bad' and the information messages indicate that records have more fields than they should, change this setting in the metadata panel and reload the data.

Specify record information

Trailer Information

Default is null

Internal file format Type

Options include:

  • AVRO
  • ORC

Qlik Catalog Stored Format Type step is skipped for single node environments where Stored Format Type is always TEXT_TAB_DELIMITED.

Specify stored file format

Record Layout Preview

Select Record Layout Preview to review and edit field columns.

Save the Entity

Information note

When saving a FlatFile source has a large number of columns (behavior seen with columns @5k), the Add Entity stage + Save Entity popup displays with loading spinner this stage can be very slow (several minutes, up to 5) and may appear to be stuck.

The following fields will populate and can be edited (column order will vary based on user configuration):

  • Name (required)
  • Index (user-defined if manually entered or auto-populated from source doc)
  • Business Description (optional)
  • Technical Description (optional)
  • Data Type (required)
  • Required: NOT NULL where a NULL value will be marked as UGLY (Default is 'false')
  • Encrypted at Source (Default is 'false')
  • Key: Primary Key (Default is 'false')
  • Validation Regex (Default is' ' [empty])
  • Foreign key (Default is 'false')

Record layout preview allows field removal or additional fields to be added

Load Data to the Entity.

Once the Entity has been saved, navigate to Entities screen in Source to load the data.

Select Load from the More dropdown.

Load data to the entity

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!