Administration of QVDs and metadata import
Qlik Data Catalyst allows users to import QVDs from a mapped instance of Qlik Sense.
QVD Ingest requires Administrators to provide information about Qlik Sense servers. Qlik Data Catalyst queries the Qlik Sense server obtaining a list of connections and corresponding paths. Qlik Data Catalyst de-dupes the list and constructs a list of unique paths. An Admin must provide a unique source name for each one of those paths. Qlik Data Catalyst stores this mapping between folders and source names for use in entity creation. Upon sync, metadata is loaded to create an entity to which data is loaded under a QVD source folder.
QVD import workflow
Administrators configure Qlik Sense Connectors and QVD Import from QVD Import screen, accessed by clicking on "Admin" in the top right of the Qlik Data Catalyst UI and selecting "QVD Import" tab.
In order to import QVDs, Admins access a data source in Qlik Sense. The following directions to import QVDs assume the environment has been configured and all prerequisites have been met. The following steps detail creation of a Connector in Qlik Data Catalyst and then import of QVDs.
Login to Qlik Data Catalyst with valid credentials. (User must have Admin privileges to access and manage Admin tab.)
Click on Admin on top right-hand side of top task bar
Click on QVD Import tab
Select Add New Connector
On the QLIK SENSE CONNECTOR panel enter:
- Connector Name: Required, User defined
- Default QVD Mount Point: Required. This value can be entered manually or selected through the file browser.
- Host: Required, Qlik Sense Host URL (ex., ducks-sense2.ad.qdcdata.net)
- Port: Optional (can be skipped)
- Username: Not currently in use (can be skipped)
- Proxy: Required, Proxy is created in Qlik SenseQMCProxy Section Note: Proxy Field is case-sensitive and connection will fail if case does not match. This is the Identification prefix entered when setting up the Virtual Proxy – "qdc" is typically used as a prefix
- QDC Base Directory: This is where Qlik Data Catalyst stores the data on local file system. (Base directory information is found here: Support->About->Settings). Copy the value from the "loadingdock.base" property: (e.g., ‘/usr/local/podium/data/’)
- Default Entity Level: REGISTERED is the only default entity level available for Qlik Data Catalyst for QVDs.
- Qlik Sense Global Unique ID: Auto-retrieved upon "Test Connection", every installation of Qlik Sense has a Globally Unique Identifier (GUID). (Note that Groups are automatically generated, named, and synced by capturing the Qlik Sense Connector Globally Unique ID which is 36 characters and removing 4 hyphens to comply with Linux Group name 32-character limit.).
- JSON Web Token (JWT): This cut-and-paste token is generated as part of Proxy set up, obtain from IT or Admin responsible for configuration.
Click on Test Connection. Upon Connection Success, Save the Connection.
Click on Show QVD Paths to set up paths
In the QVD Paths screen, click on Sync Paths. All available Qlik Sense connections that have had the "QVD Catalog" tag applied are filtered and imported into Qlik Data Catalyst. When the paths are synced, the Qlik Sense Windows folder is mapped to the Linux QDC path folder, thereby making Qlik Data Catalyst aware of each QVD in the Qlik Sense folders. Every QVD in Qlik Sense corresponds to a new QVD entity in Qlik Data Catalyst.
Click on the pencil icon on any Qlik Sense Path. Select the Linux Path using the File Browser (this is the mounted Linux path that maps to Qlik Sense Windows path). Name the Source. QDC Base Directory (will be modifiable later) and Default Entity Level will autopopulate from the Connector values. Click Ok.
Upon Sync, statuses regarding any path/folder updates display as Added, Removed, or Changed. An Admin user must select Accept to the right of each path in order to persist the path/metadata in Qlik Data Catalyst so that the application knows to scan the folder path and extract folder information.
After accepting statuses, go back to QVD Import screen. Select the Connector from Connector tabs on left and select Schedule and Run. The mounted folder (Linux path) is scanned and the QVD entities are added to Sources that have been named. (In above example TEST_DOC1, TEST_DOC2, and TEST_DOC3 are Source names.)
When Run is initiated, Qlik Data Catalyst scans the folder, finds QVDs, and creates/updates/deletes QVDs in the Source in Qlik Data Catalyst. File attributes are read from the XML Header of the source QVD in Qlik Sense, and information about the QVD required to build a metadata environment (such as Fields/Columns) for a QVD entity in Qlik Data Catalyst is extracted in this step.
Full Load: Complete build/refresh; deletes any objects and data that had been in these sources. Use Full load with caution; if sources had entities with data in them before overwrites impacting those sources are made, the data will be lost.
Re-Sync: In the case of a Re-Sync any QVDs that are no longer present in their location in the file system are deleted, including any associated data and metadata.Net-new QVDs found in the file system are ingested. QDC-owned properties/attributes that are not naturally created by the ingestion process are not populated nor have default values where defaults exist.For QVDs that exist in Qlik Data Catalyst and are still present in the file system, metadata properties/attributes owned by Qlik are refreshed/replaced but Qlik Data Catalyst-owned business metadata properties/attributes that are not naturally created by the ingestion process are retained.
Incremental Load: Use this option for the Initial Load. After the first load, all subsequent data sets loaded are added to existing data. Incremental Loads can be run One time Immediately or Custom Scheduled with entry of a Crontab Expression where a string of six or seven fields describe schedule details separated by a space (e.g., "0 15 10 15 *?$"). Enable the Crontab expression to initialize a new schedule.
Select Finish to initiate the load, re-sync, or exit the dialog in the case of scheduled future runs.
VIEW LOGS The Log screen displays upon Run for users to monitor the Connector metadata/data load.
Import of Metadata is a mechanism for importing pre-defined metadata fields and values through the use of an Excel spreadsheet. Metadata can be uploaded into environments to populate Business Metadata such as Descriptions or Tags or Relationships and Keys where information about primary and foreign keys shared between related entities can be imported from Excel Files.
Business metadata import
To update business metadata for Entities users upload an Excel (XLS) with pre-defined columns detailing Target Source, Entity, and Fields and updating metadata for Business Name, Business Description, Technical Description, Tag 1, Tag 2, and Tag 3.
*Important*: Business metadata values will be mapped to the specified object-level(s) in the spreadsheet (i.e., if Source and Entity names are provided in the spreadsheet, and the entity level property has been created, the value will be inserted at the entity property). Business Name column MUST be filled in: null values are not allowed in this column.
Select radio button for Import and upload XLS file from desktop or local system. Select Internal (Discover module) and/or External objects for update. Select Finish.
All updated business metadata now displays for the entity.
Import relationships and keys
Information about primary and foreign keys shared between related entities can be imported from Excel (XLS) files. Once imported, these relationships are displayed in several places in the UI:
In the Catalog module (on the Entity Details page)
In Field Detail screens (General Information and Lineage tabs) Discover modules
In Field grid, if User Preferences are selected to make Primary Key and/or Foreign Key visible
These relationships are populated into the metadata table pd_field_related_field
XLSX: Structure and format specification
An Excel (.XLSX) file is created and formatted manually by end users.
The file used to import relationships and keys can populate the relationship and primary/foreign key information for internal entities detailed in Discover and Catalog modules.
Validation is performed to ensure that parent and child objects exist in the target environment and to verify the relations between parent and child objects are valid.
Example: XLSX example
Note that User can ADD or REMOVE the Foreign Key relationship in the Action column.
Importing Relationships and Keys: Select IMPORT/EXPORT METADATA. Click on the radio button for Import and Select Relationships and Keys from dropdown. Choose the Excel file from laptop or local directory and select whether the relationship will be imported to external and/ or internal entities:
Upon FINISHED Job Status, users will see Primary Keys and Foreign Keys denoted in the UI.
Catalog: Related Entities Grid, the Entity containing Primary Key will be the PARENT
Internal Field General Information tab
Field General Information Tab
Field grid, if User Preferences are selected to make Primary Key and/or Foreign Key visible