Lineage Process (Bottom) Panel
Along with the pictorial graph of the lineage, you may also analyze the transformation or operations acting on the columns and tables in the lineage. This information is generally presented in the Processes Panel at the bottom of the lineage page.
Steps
- Begin a lineage trace.
- Ensure that the Process Panel, at the bottom of the page is expanded.
- Click on a column to see:
- Processes that make <> to show the operations which lead to or are upstream (source) in the lineage from the selected column.
- Processes that use <> to show the operations which use the selected column as a source for downstream lineage.
- If Limited control flow display is enabled, then go to the lineage Diagram and click on target elements and the control flow that the target depends upon will appear.
You can see
- The Process that populated Customer.ID
- Its Context
- The Data Operations within that Process which contribute to Customer.ID presented as a summary operation (not individual steps or transformations)
- Any Control Operation showing control lineage that affects Customer.ID
- An icon to Show Lineage Details for the complete Process.
Example
In the following example diagram we have the Staging DW.MITI-Finance-Staging-DW.dbo.Customer table selected and the Lineage tab selected.
The bottom of the DIAGRAM lineage display shows two columns:
- Processes that make <> - Processes that write to (make) the selected classifier
- Processes that use <> - Processes that read from (use) the selected classifier
Based upon <> which is the currently selected classifier (e.g., table), (as opposed to the point of origin for the trace which is noted by the red pin).
Thus, the left panel shows:
- A Data Mapping that writes to columns in the Customer table:
- A SQL Script in a Databricks notebook that again writes to columns in the Customer table:
We even see the operation.
On the right column we have where the Customer table is used in the lineage flow:
- A Spark Script in a Databricks notebook that reads from columns in the Customer table:
You may click on the Show lineage details icon to see detail of the Databricks notebook script:
This is no longer a lineage trace, but a lineage overview of the particular ETL/DI model that you wanted to Show lineage details for.
Clicking on any object in the flow and it will highlight that subset in the diagram and show the Processes that make and Processes that use for that selected object:
In addition, you may also see the actual code of the script using the SCRIPTS button:
Then, clicking on a step in the diagram, the matching script text is highlighted:
And vice-versa, when text is selected, the corresponding diagram object is selected:
You may return to the non script presentation by clicking PROCESSES:
The middle column in the Process (bottom) panel presents the contents (columns) of the selected object. As this diagram is not the original lineage trace, but a lineage overview of the particular ETL/DI model that you wanted to Show lineage details for, we see columns and thus the middle part of the Process (bottom) panel.
Returning back to the original lineage trace and click on Columns and selecting all of them:
We now see columns in the diagram and see the middle region in the Process (bottom) panel, because we have now downloaded and rendered the column information as part of the lineage:
You may also simply Open a Process using that icon:
In this case it is a query mapping as part of a data mapping that populates the Customer table.
Processes (Bottom) Panel with Control Flow
You may see details about control flow using the Processes Panel at the bottom of the lineage diagram in that section.