Skip to main content
Close announcements banner

Data Catalog for insight and actions

Catalog is an interactive marketplace dashboard providing immediate insight and actions for searchable and filterable QVD entities across the data ecosystem.

  • KPI framework measuring QVDs against Operational, Size, and Popularity metrics.
  • Search and drill-in to access details about Lineage, Sample Data, Creation Date, Source Connection, KPI Scoring Detail, and Field Profile Statistics.
  • Shop-for-data functionality Search, Filter, Browse, Preview targeted entity collections.

QDC for QVDs marketplace dashboard

Key Performance Indicators (KPIs)

Data KPIs are a key component of the marketplace dashboard. KPIs measure QVD entities on business and health objectives like Size, Operational effectiveness, and Popularity.

KPI calculation is performed as a post-process so does not impact the success or failure of the load or dataflow execution.

KPI Display

KPI colors provide a quick gauge of how an entity scores on the following thresholds.

KPI colors

Color

Score

Meaning

Green

68%-100%

Good or Great. Meets business objective.

Yellow

34%-67%

Risky or Ok. Needs attention/ is not performing optimally.

Red

1%-33%

Below acceptable threshold or not performing relative to other entities. Note: This may not be a meaningful score if the business value of the data source is not reflected by the KPI . For example, in the case of popularity, an entity may get a low popularity score when the components driving the score (such as number of tags or frequency of publish and or prepare jobs) are not relevant or valuable measures for that entity.

Blue (QVD only)

1-100%

Blue is used to indicate Size for QVDs; this metric does not indicate relative performance (good/bad) but simply a percentage size of the current QVD relative to the largest known QVD currently in the system.

KPI Scoring for QVDs

QVD KPI Scoring: Operation, Quality, Performance KPIs and Component Detail

Score

Component

Weight

Description/Calculaton

Operational

Recency

100%

Computes the difference between now and the last load and compares that to the typical update frequency. This metric will be N/A until QDC sees the first refresh coming from Qlik Sense.

Components

Last_Load_Date_Time=Last time the QVD was loaded on Qlik Sense

First_Load_Date_Time=Time the QVD is synced on Qlik Sense upon QVD ingestion

Typical Period (typical interval between updates)=(Last_Load_Date_Time - First_Load_Date_Time)/Number_of_Loads

Latest Period=Now - Last_Load_Date_Time

Number of Loads=Number of times the QVD was loaded on Qlik Sense since first QDC ingest

If Number_of_Loads=1 THEN RECENCY is N/A

If Latest_Period <= Typical_Period THEN Recency = 1 (100%)

If Latest_Period > Typical_Period THEN Recency =.5

If (Latest_Period - Typical_Period) / Typical_Period<=0 THEN Recency=0

Note: All variables in these calculations refer to the QVD generation on the Qlik Sense side, NOT the loads of QVD metadata into QDC.

Operational Total

100%

Size

Size Volume

50%

Relative size of the QVD in memory/storage units (file size on disk)

(QVD Size - Smallest QVD Size) / (Largest QVD Size - Smallest QVD Size)

If all QVDs are the same size (unlikely) =1

Size

Size Records

50%

Relative size of the QVD in number of records

(# records in QVD - smallest # records across all QVDs) / (largest # records across all QVDs - smallest # records across all QVDs)

If all QVDs have the same number of records (unlikely)=1

Size Total   1000% (Replaces Quality metric applied to Non-QVD entities)

Popularity

Number of Tags

15%

Count of tags compared to average (includes both tags imported from Qlik and tags created in Qlik Data Catalyst)

Average Tags=Total # of tags across all QVDs / # of QVDs

0.5 + ((# tags for the QVD - Average_Tags) / Average_Tags) *0.5)

If above is greater than 1 = 1

If Average_Tags is 0 = 0

Popularity

Number of Comments

20%

Count of comments (Qlik Data Catalyst Comments, previously known as "blog") compared to average

Average Comments=Total # of comments across all QVDs / # of QVDs

0.5 + ((# comments for the QVD - Average_Comments) / Average_Comments) *0.5)

If above is greater than 1 = 1

If Average_Comments is 0 = 0

Popularity

Number of Publish Jobs

65%

Count of Publish jobs executed including QVD compared to average

Average Publish Jobs= Total # of executed publish jobs across all QVDs / # of QVDs

0.5 + ((# executed Publish jobs using the QVD - Average_Publish_Jobs) / Average_Publish_Jobs) *0.5)

If above is greater than 1 = 1

If Average_Publish_Jobs is 0 = 0

Popularity Total

100%

Behavior and configuration of KPI metrics

If any or all metric computations fail, the data load can still complete successfully and the reason for the failure is logged. Therefore, to refresh scores, reload the entity.

Admins: KPI computation can be turned off for an entity or source by enabling the KPIs at entity and source levels or setting global core-env property: enable.kpi=true (default)

To configure a defined time window in which to capture loads, the following property can be set globally in core_env.properties(unit of measure= days):metric.computation.window=60 (default)

Note: Setting this value to a negative number (e.g.,"-1" ) will use all historical loads to compute metrics.

To configure the time period in days for which the historical custom entity metrics are retained, the following property can be set globally in core_env.properties (unit of measure=days): metric.retention.window=60(default)

Note: Setting this value to a negative number (e.g.,"-1”) will delete all historical metrics keeping only the current (latest) ones.

Catalog Page Navigation

Search, filters, and view

Tile View

Catalog tile view

Table View

Catalog table view

Catalog Tiles

Every row in table view or tile in tile view provides:

  • Entity name
  • Source name
  • View QVD Entity Details: Select to drill into detailed information about the entity and possible actions to take.
  • Add to Cart: Add this entity to your cart for dataset creation or inclusion, or publish to Qlik Sense.
  • KPI Scores: Operational, Size, Popularity.
  • Entity LevelRegistered. This is the default management level at which statistics are calculated. Analysis of registered objects produces summary statistics (profiling and validation information) and metadata about the data is retained. Sample data is available.
  • Row Count: Number of records.
  • Field Count: Number of columns.
  • Last Load: Date of last ingest or run.

Tile Details

Search

Search by name lookup searches names against individual, multiple, or all (Select All) on the following object options: data sources, QVDs, and fields are most relevant for QVDs—search can be refined to only look in the following attributes: Name, Tags, Business Name, Business Description, or Select All attributes to search. The search includes substring lookup. Note that if no name is specified, a blank query will return all objects and attributes.

Search Results for a string "person" across selected objects in catalog

Filters

Available filter and sort options display in a modal when the filters button is selected.

Filter menu options

C:\Users\User\Dropbox\filteroptions.png

Catalog Filters

Filter

Description

Datasets

Logical collections of entities as grouped in discover.

(Datasets are currently not applicable to QVDs)

Tags

Metatags applied by users in discover to assist in locating and organizing data (field, entity, source) in Qlik Data Catalyst.

Source

Parent source objects containing entity collections.

Sort by

Dropdown criteria (see Figure above):

  • Entity name: Entities that satisfy all selected criteria display by name alphabetically.
  • Field count: Entities list by order of field count.
  • Last load: Entities ordered by load time from most recent to least recent (or least recent to most recent if descending Sort is selected).
  • Operational score: Entities are ordered by composite operational score, Recency (50%) + Percent Finished Loads (50%).
  • Size score (Quality Score is not applicable to QVDs): Entities are ordered by compositeSize Volume (50%) + Size Records (50%).
  • Popularity score: Entities are ordered by composite popularity score, #Tags (15%) + #Comments (15%) + #Prepare Jobs (35%) + #Publish Jobs (35%).
  • Row count: Entities are ordered by number of records.

Sort Direction

Click arrow to change Sort Direction:

  • Up arrow: Shows ascending data, low to high values or A-Z order.
  • Down arrow: Shows descending data, high to low or Z-A order.

Entity Details

To access entity details, select the view icon on the tile or row of the entity.

Catalog tile

Entity Details: Note that in addition to entity Information and KPI scoring detail; Field Statistics and Metadata, Properties, and Related Entities accordion tabs expand and display for the QVD.

Entity Details

Add to Cart to include QVDs in a dataset or Create Dataset with selected QVDs, or Publish to Qlik Sense or Qlik Sense Advanced .

Add to Cart

Publish to Qlik Sense and Publish to Qlik Sense Advanced

Users are able to choose one or more QVDs in either catalog (by selecting cart icons on QVD tiles of interest) or discover (selecting/multi-selecting the row or rows then Add to Cart). Either method will place the QVDs in the cart from where users can Take Action to choose a destination for the selected QVD. This section will focus on the publish options.

Create or Add to an Existing Dataset: Datasets are logical collections of sources or entities. It is an organizational framework only, there are no particular rules dictating how users group the objects they have access to. Choose to Create and name a new set or Add the selected QVDs to an already existing dataset.

Publish to Qlik Sense: Exports QVDs to a preconfigured instance of Qlik Sense Enterprise. This option creates a new application for each published entity in Insight Advisor, the entry point for exploring your data and creating visualizations.

Publish to Qlik Sense Advanced: Exports QVDs to user's choice of location in Qlik Sense Enterprise: Insight Advisor (default), Data Load Editor, or Data Load Manager. Developers building Qlik Sense applications select to publish directly to Data Load Editor or Data Manager by setting a default target page in User Preferences (Profile>User Profile and Preferences>Publish to Qlik Sense Starting View) with the ability to over-ride the default setting after selecting Publish to Qlik Sense Advanced from (My Cart) action dropdown.

Developers building Qlik Sense applications can create a new application with a published entity (as before) or pick an existing Qlik Sense app from the list of existing user apps and add published entities to it.

Set Default Choice of Starting Target View in Profile

Over-ride Publish to Qlik Sense Default Starting View and optionally choose Qlik Sense application to add QVD entity to

To access a Lineage Report for the Entity, select the Lineage icon.

QVD Lineage Report

Lineage

Data lineage in Qlik Data Catalyst displays an entity's origins and exports or transforms over its lifecycle. Qlik Data Catalyst catalog supports a lineage graph display that visually details bidirectional events: downstream (descendants) and upstream (ancestor) dependencies. Users are able to control the Direction and Depth in the UI graph through display level dropdowns in the upper right of the graph. Lineage can also be accessed for individual entities through the More dropdown in the discover entity grid.

Double-click on an individual node to collapse or expand the nodes underneath (downstream of) that node.

Qlik Data Catalyst: Lineage Graph Displaying Upstream Lineage for Qlik Sense QVDs with JSON display

Lineage Graph Legend (Only relevant icons are described.)
Icon Description

Process: Blue dots represent processes that stage entities into different object types (ex. from External Source to External Entity to Internal Entity).

Available process types are:

M: Technical Metadata load

L: Data load

Qlik App: Lineage of the QVD file coming from the Qlik Installation and stored in the QVD XML header, representing how the QVD file was created. The name of the Qlik app is either the file extracted from either Windows or Linux filepath, the App Session id, or UUID (Universally Unique Identifier)—depending on how lineage information is extracted from QVD XML CreatorDoc tag.

This icon is also used to represent a Qlik App publish target.

Source: Source in this context represents the external database object as named in Qlik Data Catalyst. Entities (tables)/ fields (columns)  are selected for inclusion.

External Entity: The entity is an external representation in Qlik Data Catalyst of how the table is stored in the originating RDBMS source format. The entity is displayed with original source metadata related to ingest in Qlik Data Catalyst source module.

Internal Entity: The entity has been internalized + metadata environment is established. Once data is loaded, it is stored inreceiving directory. Internal entity metadata can be found in Qlik Data Catalyst discover module.

QVD Internal: QlikView Data file imported from Qlik Sense. QVDs are entities that behave like other source types though KPIs measures Size rather than performance

QVD External: QlikView Data file external representation as stored in Qlik Sense server database

Other: External process outside of Qlik Data Catalyst that doesn't fall under any of the other object categories

To view sample data for a QVD, select Sample Data.

Sample Data for a QVD

Field Statistics and Metadata

Expand field statistics and metadata dropdown to display Field Statistics for the entity that is being detailed. If data has not been loaded or the sample is too small, There is no data available status will display.

The field statistics that display are hard-coded and include: Field Name, Data Type, Min Value, Max Value, Percentage Nulls, Number of Nulls, Cardinality, Total Count.

Properties, Internal Entity

The properties accordion dropdown on the catalog page displays properties selected in the profile area of Qlik Data Catalyst.

Entity properties that display are configurable by selecting Profilethen Properties for Gridthen Internal Entity (expand) and select individual properties to display in Entity Detail: Properties.

If data has not been loaded, "There is no data available" status will display.

Grid Properties Selection

Selected Internal Entity properties display in Properties drop-down.

KPI Administration and Discovery

Maintenance Services in the administration section allows users to refresh metric computation and/or purge metric component calculations.

Maintenance Services

To view current KPI metric component values available for entities, navigate to discover and Select Sample Data from the entity row dropdowns, from Filter dropdown select Custom Metrics and Reload.

View Potential Metric Components