Enabling lineage for Big Data Jobs

Setting up data lineage with Atlas

You can use Apache Atlas to capture lineage for Talend Spark Jobs.

If you are using Hortonworks Data Platform V2.4 onwards to run your Jobs and Apache Atlas is already installed in your Hortonworks cluster, you can make use of Atlas to trace the lineage of given data flow to discover how this data was generated by a Spark Job, including the components used in this Job and the schema changes between the components.

If you are using CDP Private Cloud Base or CDP Public Cloud to run your Jobs and Apache Atlas is already installed in your cluster, you can also make use of Atlas either using Spark Universal 3.3.x (recommended) or Cloudera CDP 7.x [Built-in] in Spark Configuration.

1. Click Run to open its view and then click the Spark configuration tab.
2. From the Distribution list and the Version list, select your Hortonworks distribution. The Use Atlas check box is displayed.

   With this option activated, you need to set the following parameters:

   • Atlas URL: Enter the address of your Atlas instance, such as http://name_of_your_atlas_node:port.

   • In the Username and Password fields, enter the authentication information for access to Atlas.

   • Set Atlas configuration folder: Select this check box if your Atlas cluster uses custom properties like SSL or read timeout. In the field that appears, enter the path to a local directory containing your atlas-application.properties file. Your Job will then use these custom properties.

     Ask the administrator of your cluster for this configuration file. For more information, see the Client Configs section in Atlas configuration.

   • Die on error: select this check box to stop the Job execution when Atlas-related issues occur, such as connection issues to Atlas. Otherwise, leave it clear to allow your Job to continue to run.

After you configure the connection and run the Job, lineage is generated in Atlas. You still need to configure the other parameters in the Spark configuration tab in order to successfully run the Job. For more information, see Creating Spark Batch Jobs.

After execution, search in Atlas for the lineage information written by this Job.

Reading the Atlas lineage

In Atlas, lineage for a Job consists of two entity types:

• The Job entity.

• Component entities that use data schemas, such as tRowGenerator or tSortRow. Connection or configuration components such as tHDFSConfiguration are not included because they do not use schemas.

The example Job generates six entities (one Job and five components) and automatically adds these tags:

• Talend for all entities generated by the Job.

• TalendComponent for component entities.

• TalendJob for Job entities.

You can directly click one of these tags in Atlas to display the corresponding entities.

For example, you can click TalendJob to show a list of Job entities logged in Atlas and select a Job to see the lineage information of this Job. The following image shows how data flows through the example Job.