Skip to main content
Close announcements banner

Administration of QVDs and metadata import

QVD import

Qlik Data Catalyst allows users to import QVDs from a mapped instance of Qlik Sense.

QVD Ingest requires administrators to provide information about Qlik Sense servers. Qlik Data Catalyst queries the Qlik Sense server obtaining a list of connections and corresponding paths. Qlik Data Catalyst de-dupes the list and constructs a list of unique paths. An admin must provide a unique source name for each one of those paths. Qlik Data Catalyst stores this mapping between folders and source names for use in entity creation. Upon sync, metadata is loaded to create an entity to which data is loaded under a QVD source folder.

QVD Import Tab

QVD import workflow

Administrators configure Qlik Sense Connectors and QVD Import from QVD Import tab, accessed by clicking on Admin in the top right of the Qlik Data Catalyst UI and selecting the tab. 

In order to import QVDs, Admins access a data source in Qlik Sense. The following directions to import QVDs assume the environment has been configured and all prerequisites have been met. The following steps detail creation of a Connector in Qlik Data Catalyst and then import of QVDs through the Connector.

Login to Qlik Data Catalyst with valid credentials. (User must have Admin privileges to access and manage Admin tabs.)

Click on Admin on top right-hand side of top task bar, select QVD Import tab.

Select Add New Connector

On the QLIK SENSE CONNECTOR panel enter:

  • Connector Name: Required, user defined.
  • Default QVD Mount Point: Required. This value can be entered manually or selected through the file browser. [This value is specified in core_env property: localfile.base.dir.source.connection]
  • Host: Required, Qlik Sense host URL (ex., ducks-sense2.ad.qdcdata.net)
  • Port: Optional (can be skipped)
  • Username: Not currently in use (can be skipped)
  • Proxy: Required, proxy is created in Qlik Sense>QMC>Proxy Section. Note: Proxy field is case-sensitive and connection will fail if case does not match. This is the Identification prefix entered when setting up the Virtual Proxy – "qdc" is typically used as a prefix.
  • QDC Base Directory: This is where Qlik Data Catalyst stores the data on local file system. (Base directory information is found here: Support>About>Settings). Copy the value from loadingdock.base property: (e.g., /usr/local/podium/data/)
  • Default Entity Level: REGISTERED is the only entity level available for Qlik Data Catalyst for QVDs.
  • Qlik Sense Global Unique ID: Auto-retrieved upon Test Connection, every installation of Qlik Sense has a Globally Unique Identifier (GUID). Note that groups are automatically generated, named, and synced by capturing the Qlik Sense Connector Globally Unique ID (GUID) [which is 36 characters; four hyphens are removed to comply with linux group name 32-character limit].
  • JSON Web Token (JWT): This cut-and-paste token is generated as part of proxy set up. Obtain this token from IT or the administrator responsible for configuration.

Test Connection. Upon Connection Success, Save the connection.

Click on Show QVD Paths to set up paths

In the QVD Paths screen, click on Sync Paths. All available Qlik Sense connections with the QVD Catalog tag applied are filtered and imported into Qlik Data Catalyst. When the paths are synced, the Qlik Sense Windows folder is mapped to the Linux QDC path folder, thereby making Qlik Data Catalyst aware of each QVD in Qlik Sense folders. Every QVD in Qlik Sense corresponds to a new QVD entity in Qlik Data Catalyst.

Click on the pencil icon on any Qlik Sense Path. Select the Linux Path using the file browser (this is the mounted Linux path that maps to Qlik Sense Windows path).   Provide the Source Name. QDC Base Directory (will be modifiable later) and Default Entity Level (always REGISTERED) will auto-populate from the connector values. Click Ok.

Upon sync, statuses regarding any path and folder updates display as Added(green dot), Removed(red dot), or Changed(yellow dot). An Admin user must select Accept to the right of each path in order to persist the path metadata in Qlik Data Catalyst so that the application knows to scan the folder path and extract folder information.

After accepting statuses, go back to QVD Import screen tab. Select the connector from connector tabs on left and from header bar select Schedule and then Run. The mounted folder (Linux path) is scanned and the QVD entities are added to sources that have been named. (In above example TEST_DOC1, TEST_DOC2, and TEST_DOC3 are Source Names.)

Ensure that Window and Linux path endpoint values match

When Run is initiated, Qlik Data Catalyst scans the folder, finds QVDs, and create or updates or deletes QVDs in the sources in Qlik Data Catalyst. File attributes are read from the XML header of the source QVD in Qlik Sense, and information about the QVD required to build a metadata environment (such as fields and columns) is extracted in this step.

Three load types are available:

Full Load: Complete build and refresh; deletes any objects and data that had been in these sources. Use Full load with caution; if sources had entities with data in them before a Full Load overwrites them and the data will be lost.

Re-Sync: In the case of a Re-Sync any QVDs that are no longer present in their location in the file system are deleted, including any associated data and metadata.Net-new QVDs found in the file system are ingested. QDC-owned properties and attributes that are not naturally created by the ingestion process are not re-populated nor will they have default values where defaults exist. For QVDs that exist in Qlik Data Catalyst and are still present in the file system, metadata properties or attributes owned by Qlik are refreshed but Qlik Data Catalyst-owned business metadata properties and attributes that are not naturally created by the ingestion process are not retained.

Incremental Load: Use this option for the Initial Load. After the first load, all subsequent data sets loaded are added to existing data. Incremental Loads can be run One time Immediately or Custom Scheduled with entry of a Crontab Expression where a string of six or seven fields describe schedule details separated by a space (e.g., "0 15 10 15 *?$"). Enable the Crontab expression to initialize a new schedule.

Select Finish to initiate the load, re-sync, or exit the dialog in the case of scheduled future runs.

VIEW LOGS The Log screen displays upon Run for users to monitor the Connector metadata or data load.

Metadata import

Import of Metadata is a mechanism for importing pre-defined metadata fields and values through the use of an Excel spreadsheet. Metadata can be uploaded into environments to populate Business Metadata such as Descriptions or Tags or Relationships and Keys where information about primary and foreign keys shared between related entities can be imported from Excel Files.

Business metadata import

To update business metadata for Entities users upload an Excel (XLS) with pre-defined columns detailing Target Source, Entity, and Fields and updating metadata for Business Name, Business Description, Technical Description, Tag 1, Tag 2, and Tag 3.

*Important*: Business metadata values will be mapped to the specified object-level(s) in the spreadsheet (i.e., if Source and Entity names are provided in the spreadsheet, and the entity level property has been created, the value will be inserted at the entity property). Business Name column MUST be filled in: null values are not allowed in this column.

Validate and Save

Screen Shot 12 (2)

Select radio button for Import and upload XLS file from desktop or local system. Select Internal (Discover module) and/or External objects for update. Select Finish.

Import business metadata

All updated business metadata now displays for the entity.

Updated business metadata

C:\Users\User\Dropbox\Screen Shot 2019-05-02 at 6.04.26 PM.png    

Import relationships and keys

Information about primary and foreign keys shared between related entities can be imported from Excel (XLS) files. Once imported, these relationships are displayed in several places in the UI:

  • In the Catalog module (on the Entity Details page)

  • In Field Detail screens (General Information and Lineage tabs) Discover modules

  • In Field grid, if User Preferences are selected to make Primary Key and/or Foreign Key visible

These relationships are populated into the metadata table pd_field_related_field

XLSX: Structure and format specification

An Excel (.XLSX) file is created and formatted manually by end users.

The file used to import relationships and keys can populate the relationship and primary and foreign key information for internal entities detailed in discover and catalog modules.

Validation is performed to ensure that parent and child objects exist in the target environment and to verify the relations between parent and child objects are valid.

Example: XLSX example

Note that User can ADD or REMOVE the Foreign Key relationship in the Action column.

XLSX example

Importing Relationships and Keys: Select Import/Export Metadata. Click on the radio button for Import and Select Relationships and Keys from dropdown. Choose the Excel file from laptop or local directory and select whether the relationship will be imported to external and or internal entities:

Import Export Relationships and Keys

Upon FINISHED Job Status, users will see Primary Keys and Foreign Keys denoted in the UI.

Import Export Metadata Job Status

Catalog: Related Entities Grid, the Entity containing Primary Key will be the PARENT

Catalog Related Entities

Internal Field General Information tab

Field General Information Tab

Field grid, if User Preferences are selected to make Primary Key and/or Foreign Key visible

Field and External Field Grids