QlikView Data (QVD) file import

Qlik Data Catalyst allows users to import QVDs from a mapped instance of Qlik Sense. The process of QVD Ingest requires Administrators to provide information about Qlik Sense servers inside Qlik Data Catalyst (Admin | QVD Import). Qlik Data Catalyst queries the server and gets a list of connections and corresponding paths. Qlik Data Catalyst dedupes the list and constructs a list of unique paths and the Admin must provide a unique source name for each one of those paths. Qlik Data Catalyst stores this mapping between folders and source names in its metadata for use in entity creation.

QVD import workflow

In order to import QVDs, Admins access a data source in Qlik Sense. The following directions to import QVDs assume the environment has been configured per "QlikDataCatalystQlikSenseIntegration_QVDImport" and all prerequisites have been met.

If needed, refer to Qlik Sense documentation to create a connection and add data: Creating a connection and selecting data

The following steps detail creation of a Connector in Qlik Data Catalyst and import of QVDs.

1. Login to Qlik Data Catalyst with valid credentials. User must have Admin privileges to access and manage Admin tab.
2. Click on Admin on top right-hand side of top task bar
3. Click on QVD Import tab

4. Select Add New Connector

   

5. On the QLIK SENSE CONNECTOR panel enter:

   • Connector Name: Required, User defined
   • Default QVD Mount Point: Required. This value can be entered manually or selected through the file browser. [This value is specified in core_env property: localfile.base.dir.source.connection]
   • Host: Required, Qlik Sense Host URL (ex., ducks-sense2.ad.qdcdata.net)
   • Port: Optional (can be skipped)
   • Username: Not currently in use (can be skipped)

   • Proxy: Required, Proxy is created in Qlik SenseàQMCàProxy Section

     Information note

     Proxy Field is case-sensitive and connection will fail if case does not match. This is the Identification prefix entered when setting up the Virtual Proxy – "qdc" is typically used as a prefix

   • QDC Base Directory: This is where Qlik Data Catalyst stores the data on local file system. (Copy the base directory information from Support->About->Settings). Copy the value from the "loadingdock.base" property: (e.g., ‘/usr/local/podium/data/’)
   • Default Entity Level: Populated from System Settings and can be overwritten (options are MANAGED or REGISTERED)
   • Qlik Sense Global Unique ID: Auto-retrieved upon "Test Connection", every installation of Qlik Sense has a Globally Unique Identifier (GUID). Note that Security Groups are automatically generated, named, and synced by capturing this Qlik Sense Connector Globally Unique ID which is 36 characters and removing the hyphens to comply with Linux Group name 32-character limit.)
   • JSON Web Token (JWT): This cut-and-paste token is generated as part of Proxy set up, obtain from IT or Admin responsible for configuration.
6. Click on Test Connection. Upon Connection Success, Save the Connection
7. Click on Show QVD Paths to set up paths

8. In the QVD Paths screen, click on Sync Paths. All available Qlik Sense connections that have had the "QVD Catalog" tag applied during configuration in Qlik Sense are filtered and imported into Qlik Data Catalyst. When the paths are synced, the Qlik Sense Windows folder must be mapped to the Linux path folder, thereby making Qlik Data Catalyst aware of each QVD in these folders. Every QVD in Qlik Sense corresponds to a new QVD entity in Qlik Data Catalyst.

   

9. Click on the pencil icon on any Qlik Sense Path.

   Select the Linux Path using the File Browser (this is the mounted Linux path that maps to Windows path counterpart), then name the Source. QDC Base Directory and Default Entity Level will auto-populate from the Connector values but can be overwritten.

   Stored File Format is fixed as TEXT_TAB_DELIMITED in Single Node environments. However, in Multi-Node environments, storage options include: AVRO, ORC, ORC_ALL_STRING, PARQUET, and PARQUET_ALL_STRING.

   Click Ok. *Note: Be sure that the Linux Path is the mount point for the QVD Windows Path network share which contains the QVD files on the Qlik Sense server. *

   

10. Upon Sync, status regarding any path/folder updates display as Added, Removed, or Changed.

    Accept the path to save the metadata. An Admin user must select Accept to the right of each path in order to persist the path/metadata in Qlik Data Catalyst so that the application knows to scan the folder path and extract folder information.

    
    
    

11. Go back to QVD Import page. Open the Connector and select Schedule and Run. The mounted folder (Linux path) is scanned and the QVD entities are added to the created source ("Demo_Source" in pictured example) created on the Paths screen. When Run is initiated, Qlik Data Catalyst scans the folder, finds QVDs, and creates/updates/deletes QVDs in the Source in Qlik Data Catalyst. File attributes are read from the XML Header of the originating source QVD, and information about the QVD required to build a metadata environment (such as Fields/Columns) for a QVD entity in Qlik Data Catalyst is extracted in this step.

     

    QVD Scan Functionality by Type: Full Reload, Re-sync, Incremental

    QVD scan type	Processes ONLY files updated since the last file scan	Processes all QVD files on disk (ignores scan history)	Deletes entity metadata	Merges entity metadata*	Deletes load data	Deletes load logs	Deletes previous file scan history	Loads data (when Load check box is selected)
    Full Reload		X	X️		X	X	X	X
    Re-sync		X		X	X	X	X	X
    Incremental	X							X

    Information note

    *On an entity metadata merge, the entity is updated with the latest version of the QVD's metadata, but metadata entered in the Catalyst UI (like tags and business description) is retained.

    Full Reload: Complete build/refresh; deletes any objects and data that had been in these sources. Use Full load with caution; if sources had entities with data in them before overwrites impacting those sources are made, the data will be lost. All history, logs, and information related to connectors are all deleted.

    Re-Sync: Syncs Qlik Data Catalyst with Qlik Sense. QVDs that are not found upon sync with Qlik Sense are deleted, including any associated data and metadata. Net-new QVDs found in the file system are ingested. For QVDs that exist in Qlik Data Catalyst and are still present in the Qlik Sense file system, metadata properties/attributes owned by Qlik are refreshed/replaced.

    Incremental Load: (recommended for Initial Load) After the first load all subsequent data sets loaded are added to existing data if Load Data is checked. Incremental Loads can be run One time Immediately or Custom Scheduled with entry of a Crontab Expression where a string of six or seven fields describe schedule details separated by a space (e.g., "0 15 10 15 *?$"). Enable the Crontab expression to initialize a new schedule.

    Select Finish to initiate the load, re-sync, or exit the dialog in the case of scheduled future runs. Make sure to check 'Load Data' to onboard data for the QVDs.

    

    Upon Run, Users are transitioned to Logs screen where progress of the load can be monitored. Note that when source data load is initiated from this screen, all entity data is loaded. In single and multi-node environments, if the Load Data option is not checked, users are able to load entity data individually from the source module.