Skip to main content

Lineage

Data lineage in Qlik Catalog displays an entity's origins and exports/transforms over its lifecycle. Qlik Catalog catalog supports a lineage graph display that visually details bidirectional events: downstream (descendants) and upstream (ancestor) dependencies. Users are able to control the Direction and Depth in the UI graph through Display Levels dropdowns in the upper right of the graph; these parameters can also be defined in API calls that describe the lineage graph in JSON format. Lineage can also be seen from source and discover entity grids by selecting Lineage from the More dropdown in each entity row.

Qlik Catalog: Lineage graph displaying upstream activity for a Prepare target

Example of an entity with multiple ancestors

Qlik Catalog: Lineage graph displaying upstream lineage for Qlik Sense QVD files with JSON display

Hover over any lineage icon for detailed JSON metadata

Qlik Catalog: Lineage graph legend

Icon

Description

icon lineage process

Process: Blue dots represent processes that stage entities into different object types (ex. from external source to external entity to internal entity).

Available process types are:

M: Technical metadata load

L: Data load

DF: Prepare dataFlow

P: Publish

R: Qlik Replicate

icon lineage Qlik app

Qlik App: Lineage of the QVD file coming from the Qlik installation and stored in the QVD XML header, representing how the QVD file was created. The name of the Qlik App is either the file extracted from either Windows or Linux filepath, the App Session id, or UUID (Universally Unique Identifier)—depending on how lineage information is extracted from the QVD CreatorDoc tag.

 

This icon is also used to represent a Qlik App publish target, in which case it is called Qlik Sense App by default.

 

icon lineage source

Source: Source in this context represents the external database object as named in Qlik Catalog. Entities (tables)/ Fields (columns)  are selected for inclusion.

 

icon lineage Qlik replicate node

Qlik Replicate: Qlik Data Integration (QDI) source type where metadata is synced across products. Drill in to view metadata and lineage of replicated data source. Qlik Replicate is a high-performance data replication for SQL Server databases.

 

icon lineage prepare target

Prepare Target: Entity is created as the result of a dataflow where controls (Pig transforms) and functions are applied to source entities.

icon lineage QCS target

QCS Target: Entity data has been pushed specifically to Qlik Cloud S3 based landing zone

 

icon lineage publish target

Publish Target: Entity data has been published by Qlik Catalog

 

icon lineage explore view

Explore View: These views are entities created within Qlik Catalogfrom explore canvas; fields can be selected through shopping for data or query language for creation of internal custom views.

 

icon lineage external entity

External Entity: The entity is an external representation in Qlik Catalog of how the table is stored in the originating RDBMS source format. The entity is displayed with original source metadata related to ingest in Qlik Catalog source module.

 

icon lineage internal entity

Internal Entity: The entity has been internalized + metadata environment is established. Once data is loaded, it is stored in the receiving directory. Internal entity metadata can be found in Qlik Catalog discover module.

 

icon lineage internal qvd entity

QVD Internal: QlikView Data file imported from Qlik Sense. QVDs are entities that behave like other source types though KPIs measure size rather than performance.

 

icon lineage external qvd entity

QVD External: QlikView Data file external representation as stored in Qlik Sense server database.

 

icon lineage external process

Other: External process outside of Qlik Catalog that doesn't fall under any of the other object categories

The graphs display ancestors: predecessor objects the entity is created from. Examples include sources that an entity belongs to or the sources or entities a prepare target was originally transformed from. Also shown are descendants (objects downstream that were created by causal events involving the primary entity) such as objects created from prepare transforms. The graphs are composed of nodes and edges. Nodes are Qlik Catalog objects (source, entity, dataflow, publish_job, or publish_target) and edges connect them (load, explore, prepare_dataflow, publish_job).

Single-click on a node to reveal technical metadata for sources, entities, QVDs, or access run information for prepare and publish targets, explore views.

Double-click on an individual node to collapse or expand the nodes underneath (downstream of) that node

Click on a lineage icon to reveal metadata about that process or object.

Qlik Catalog lineage UI and API

Supported APIs available in Swagger documentation (found in Support dropdown in top navigation bar in Qlik Catalogrepresent the same information that is detailed graphically in the UI as JSON listing the nodes and edges with various parameters that control the direction, depth, and the option whether to display Prepare dataflows and publish jobs as nodes rather than edges in the JSON response.

Levels of ancestors and descendants displayed is controlled by a core_env property which can be overridden at the entity level in the UI and for API requests through Swagger:

lineage.depth.default=5

The value represents the number of lineage edges displayed starting from the initial starting node, this depth goes in both directions.

GET /lineage/v1/graphs

GET /lineage/v1/nodes

The following Lineage graph captures upstream and downstream lineage for a Prepare target that is then published.

­­­­lineage graph represents same information as the following JSON about entity: reunited_pd_schema_version entity

Lineage API JSON

 

Lineage Graphs API response for entity "reunited_pd_schema_version"

This request returns edge and node metadata: Prepare Targets, Publish Targets, Dataflows associated with the entity (reference entity id: 2153]

API Request: http://<podium_host>:8080/podium-trunk/lineage/v1/graphs?referenceType=entity&referenceId=2153&direction=both&depth=5&processesAsNodes=true

API Response:

{

  "nodes": [

    {

      "id": "2169",

      "label": "lin3",

      "reference": {

        "type": "dataflow",

        "id": "3240"

      },

      "metadata": {}

    },

    {

      "id": "2170",

      "label": "reunited_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2153"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2166",

      "label": "lin2",

      "reference": {

        "type": "dataflow",

        "id": "3235"

      },

      "metadata": {}

    },

    {

      "id": "2167",

      "label": "fast_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2151"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2168",

      "label": "slow_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2152"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2164",

      "label": "Lin1",

      "reference": {

        "type": "dataflow",

        "id": "3231"

      },

      "metadata": {}

    },

    {

      "id": "2165",

      "label": "sorted_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2150"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2171",

      "label": "lin4_publish",

      "reference": {

        "type": "publish_job",

        "id": "17"

      },

      "metadata": {}

    },

    {

      "id": "12",

      "label": "local",

      "reference": {

        "type": "publish_target",

        "id": "1"

      },

      "metadata": {

        "subtype": "LOCALFILE"

      }

    }

  ],

  "edges": [

    {

      "source": "2169",

      "target": "2170",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2166",

      "target": "2167",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2166",

      "target": "2168",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2167",

      "target": "2169",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2168",

      "target": "2169",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2164",

      "target": "2165",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2165",

      "target": "2166",

      "type": "prepare_dataflow",

      "metadata": {}

    },

    {

      "source": "2171",

      "target": "12",

      "type": "export_job",

      "metadata": {}

    },

    {

      "source": "2170",

      "target": "2171",

      "type": "export_job",

      "metadata": {}

    }

  ]

}

 

 

Lineage Nodes API response for entity "reunited_pd_schema_version"

This request returns just the node metadata: Prepare Targets, Publish Targets, Dataflows that associated with the entity (reference entity id: 2153]

API Request: http://<podium_host>:8080/podium-trunk/lineage/v1/nodes?referenceType=entity&referenceId=2153&direction=both&processesAsNodes=true

API Response:

{

  "nodes": [

    {

      "id": "2169",

      "label": "lin3",

      "reference": {

        "type": "dataflow",

        "id": "3240"

      },

      "metadata": {}

    },

    {

      "id": "2166",

      "label": "lin2",

      "reference": {

        "type": "dataflow",

        "id": "3235"

      },

      "metadata": {}

    },

    {

      "id": "2167",

      "label": "fast_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2151"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2168",

      "label": "slow_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2152"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2164",

      "label": "Lin1",

      "reference": {

        "type": "dataflow",

        "id": "3231"

      },

      "metadata": {}

    },

    {

      "id": "2165",

      "label": "sorted_pd_schema_version",

      "reference": {

        "type": "entity",

        "id": "2150"

      },

      "metadata": {

        "subtype": "PREPARE_TARGET"

      }

    },

    {

      "id": "2171",

      "label": "lin4_publish",

      "reference": {

        "type": "publish_job",

        "id": "17"

      },

      "metadata": {}

    },

    {

      "id": "12",

      "label": "local",

      "reference": {

        "type": "publish_target",

        "id": "1"

      },

      "metadata": {

        "subtype": "LOCALFILE"

      }

    }

  ]

}

 

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – let us know how we can improve!