Lineage
Data lineage in Qlik Catalog displays an entity's origins and exports/transforms over its lifecycle. Qlik Catalog catalog supports a lineage graph display that visually details bidirectional events: downstream (descendants) and upstream (ancestor) dependencies. Users are able to control the Direction and Depth in the UI graph through Display Levels dropdowns in the upper right of the graph; these parameters can also be defined in API calls that describe the lineage graph in JSON format. Lineage can also be seen from source and discover entity grids by selecting Lineage from the More dropdown in each entity row.
|
Icon |
Description |
|---|---|
|
|
Process: Blue dots represent processes that stage entities into different object types (ex. from external source to external entity to internal entity). Available process types are: M: Technical metadata load L: Data load DF: Prepare dataFlow P: Publish R: Qlik Replicate |
|
|
Qlik App: Lineage of the QVD file coming from the Qlik installation and stored in the QVD XML header, representing how the QVD file was created. The name of the Qlik App is either the file extracted from either Windows or Linux filepath, the App Session id, or UUID (Universally Unique Identifier)—depending on how lineage information is extracted from the QVD CreatorDoc tag.
This icon is also used to represent a Qlik App publish target, in which case it is called Qlik Sense App by default. |
|
|
Source: Source in this context represents the external database object as named in Qlik Catalog. Entities (tables)/ Fields (columns) are selected for inclusion. |
|
|
Qlik Replicate: Qlik Data Integration (QDI) source type where metadata is synced across products. Drill in to view metadata and lineage of replicated data source. Qlik Replicate is a high-performance data replication for SQL Server databases. |
|
|
Prepare Target: Entity is created as the result of a dataflow where controls (Pig transforms) and functions are applied to source entities. |
|
|
QCS Target: Entity data has been pushed specifically to Qlik Cloud S3 based landing zone |
|
|
Publish Target: Entity data has been published by Qlik Catalog |
|
|
Explore View: These views are entities created within Qlik Catalogfrom explore canvas; fields can be selected through shopping for data or query language for creation of internal custom views. |
|
|
External Entity: The entity is an external representation in Qlik Catalog of how the table is stored in the originating RDBMS source format. The entity is displayed with original source metadata related to ingest in Qlik Catalog source module. |
|
|
Internal Entity: The entity has been internalized + metadata environment is established. Once data is loaded, it is stored in the receiving directory. Internal entity metadata can be found in Qlik Catalog discover module. |
|
|
QVD Internal: QlikView Data file imported from Qlik Sense. QVDs are entities that behave like other source types though KPIs measure size rather than performance. |
|
|
QVD External: QlikView Data file external representation as stored in Qlik Sense server database. |
|
|
Other: External process outside of Qlik Catalog that doesn't fall under any of the other object categories |
The graphs display ancestors: predecessor objects the entity is created from. Examples include sources that an entity belongs to or the sources or entities a prepare target was originally transformed from. Also shown are descendants (objects downstream that were created by causal events involving the primary entity) such as objects created from prepare transforms. The graphs are composed of nodes and edges. Nodes are Qlik Catalog objects (source, entity, dataflow, publish_job, or publish_target) and edges connect them (load, explore, prepare_dataflow, publish_job).
Single-click on a node to reveal technical metadata for sources, entities, QVDs, or access run information for prepare and publish targets, explore views.
Qlik Catalog lineage UI and API
Supported APIs available in Swagger documentation (found in Support dropdown in top navigation bar in Qlik Catalogrepresent the same information that is detailed graphically in the UI as JSON listing the nodes and edges with various parameters that control the direction, depth, and the option whether to display Prepare dataflows and publish jobs as nodes rather than edges in the JSON response.
Levels of ancestors and descendants displayed is controlled by a core_env property which can be overridden at the entity level in the UI and for API requests through Swagger:
lineage.depth.default=5
The value represents the number of lineage edges displayed starting from the initial starting node, this depth goes in both directions.
GET /lineage/v1/graphs
GET /lineage/v1/nodes
The following Lineage graph captures upstream and downstream lineage for a Prepare target that is then published.
Lineage API JSON
Lineage Graphs API response for entity "reunited_pd_schema_version"
This request returns edge and node metadata: Prepare Targets, Publish Targets, Dataflows associated with the entity (reference entity id: 2153]
API Request: http://<podium_host>:8080/podium-trunk/lineage/v1/graphs?referenceType=entity&referenceId=2153&direction=both&depth=5&processesAsNodes=true
API Response:
{
"nodes": [
{
"id": "2169",
"label": "lin3",
"reference": {
"type": "dataflow",
"id": "3240"
},
"metadata": {}
},
{
"id": "2170",
"label": "reunited_pd_schema_version",
"reference": {
"type": "entity",
"id": "2153"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2166",
"label": "lin2",
"reference": {
"type": "dataflow",
"id": "3235"
},
"metadata": {}
},
{
"id": "2167",
"label": "fast_pd_schema_version",
"reference": {
"type": "entity",
"id": "2151"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2168",
"label": "slow_pd_schema_version",
"reference": {
"type": "entity",
"id": "2152"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2164",
"label": "Lin1",
"reference": {
"type": "dataflow",
"id": "3231"
},
"metadata": {}
},
{
"id": "2165",
"label": "sorted_pd_schema_version",
"reference": {
"type": "entity",
"id": "2150"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2171",
"label": "lin4_publish",
"reference": {
"type": "publish_job",
"id": "17"
},
"metadata": {}
},
{
"id": "12",
"label": "local",
"reference": {
"type": "publish_target",
"id": "1"
},
"metadata": {
"subtype": "LOCALFILE"
}
}
],
"edges": [
{
"source": "2169",
"target": "2170",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2166",
"target": "2167",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2166",
"target": "2168",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2167",
"target": "2169",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2168",
"target": "2169",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2164",
"target": "2165",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2165",
"target": "2166",
"type": "prepare_dataflow",
"metadata": {}
},
{
"source": "2171",
"target": "12",
"type": "export_job",
"metadata": {}
},
{
"source": "2170",
"target": "2171",
"type": "export_job",
"metadata": {}
}
]
}
Lineage Nodes API response for entity "reunited_pd_schema_version"
This request returns just the node metadata: Prepare Targets, Publish Targets, Dataflows that associated with the entity (reference entity id: 2153]
API Request: http://<podium_host>:8080/podium-trunk/lineage/v1/nodes?referenceType=entity&referenceId=2153&direction=both&processesAsNodes=true
API Response:
{
"nodes": [
{
"id": "2169",
"label": "lin3",
"reference": {
"type": "dataflow",
"id": "3240"
},
"metadata": {}
},
{
"id": "2166",
"label": "lin2",
"reference": {
"type": "dataflow",
"id": "3235"
},
"metadata": {}
},
{
"id": "2167",
"label": "fast_pd_schema_version",
"reference": {
"type": "entity",
"id": "2151"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2168",
"label": "slow_pd_schema_version",
"reference": {
"type": "entity",
"id": "2152"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2164",
"label": "Lin1",
"reference": {
"type": "dataflow",
"id": "3231"
},
"metadata": {}
},
{
"id": "2165",
"label": "sorted_pd_schema_version",
"reference": {
"type": "entity",
"id": "2150"
},
"metadata": {
"subtype": "PREPARE_TARGET"
}
},
{
"id": "2171",
"label": "lin4_publish",
"reference": {
"type": "publish_job",
"id": "17"
},
"metadata": {}
},
{
"id": "12",
"label": "local",
"reference": {
"type": "publish_target",
"id": "1"
},
"metadata": {
"subtype": "LOCALFILE"
}
}
]
}