Lineage Flow Type
The Type in the upper left of the lineage display provides a selection between either:
- Lineage Trace
- DATA FLOW - Based upon connection definitions to data stores and physical transformation rules which transform and move the data)
- SEMANTIC FLOW - Based upon the definition and usage type relationships from a term, concept or logical Model to a physical representation.
- You may save a lineage graph to be shared and referred to later. This reduces the time required to read from the database and regenerate a lineage graph for larger diagrams.
You may also pick a saved lineage diagram, which someone may send to you in order to see the same presentation they did but not re-calculated, and thus rendered very fast:
-These are saved system wide and thus anyone can see them.
Lineage Overview for Models
Data Integration and ETL/ETL data processes contain lineage within the model, even without stitching them to other models. In addition, data store models such as databases with views and/or stored procedures also present lineage in this fashion.
You may even use the lineage overview on models not in the current configuration using the MANAGE > Repository function.
The Data Flow Overview lineage (see TYPE in the upper right of diagram) has a very limited scope, only to the specific subset that is a self-contained model, e.g., a schema model in a database model or a transformation model or connection model in an ETL/DI model or BI model. Thus, it has none of the information determined in an Impact or Lineage trace diagram, so that connections are not resolved (may just be a * because of “Select *” in the connection definition). To get a true lineage picture you must use the Impact or Lineage trace from a table or column (field).
In addition, in the Overview lineage of a connection model in an ETL/DI model or BI model, many of the object are not even included in an end-to-end lineage trace and thus you will not be able to trace from that object at all. This limitation is caused by the fact that connection definitions themselves only show in the overview and are not a part of a lineage trace and thus there is no way to trace lineage from them.
The data flow overview lineage presents detailed transformation lineage vs. a lineage trace which presents summary lineage.
In particular, when you select a runtime job, and go to the lineage tab, you see the detailed transformation lineage: every transformation is being depicted on the screen. This view is good as long as you only look at one job at a time.
Finally, the Data Flow presentation for overview lineage does notoffer the Tree tab as the scope is only the currently model and the Tree tab features end-to-end lineage, which is not available for overview lineage presentations.
Steps
- Sign in as a user which has at least the Metadata Viewing or Data Managementcapability object role assignment to the model to be analyzed for Overview lineage.
- Open the object page of the model (e.g., ETL/DI or BI model)
- Go to the Lineage tab.
Overview lineage – This is not a Type, per se. Instead, it describes another way to view lineage as a whole, where one scopes down to a particular model. Based upon a view of the design level lineage limited to the scope of the model you invoked it on (by clicking on the Lineage tab) and thus is not a complete end-to-end lineage picture (with the exception of the EXTERNAL DATA FLOW option), but simply an overview of the model lineage picture.
The Type may be:
- INTERNAL DATA FLOW – Design level data flow internal to the model
- EXTERNAL DATA FLOW – End-to-end data flow including other models which are stitched to lineage objects in the current model
- TRANSFORMATION DATA FLOW – Internal run time data flow which includes intermediate transformations as objects in the lineage
- SUMMARY DATA FLOW - Internal run time data flow which DOES NOT include intermediate transformations and other objects in the lineage.
You are automatically given the Overview lineage when you start by opening a model and going to the Lineage tab at the model or schema level (not for a specific table or column). In addition, only dataflow options are presented (not semantic lineage). In this case, the options above are presented.
Example
Sign in as Administrator and go to the object page for the Staging to Dimensional Talend DI model and go to the Lineage tab.
Right-click on the top DI process name ShippingPOC and select Open.
