Skip to main content Skip to complementary content

Enabling runtime lineage for Jobs

Talend Studio allows you to enable runtime lineage for Standard Jobs, which can be leveraged in a future release by the analysis capability of Talend Data Catalog for the runtime metadata, for example, the query with variables, the schema with dynamic columns, etc.

When executing a Standard Job for which runtime lineage is enabled, the information needed by Talend Data Catalog, for example, the Job name, the component name, the schema, the query, etc., will be written into a JSON file.

Information noteNote: To fully use this feature, you must install Talend Data Catalog.

About this task

To enable runtime lineage for Standard Jobs, complete the following:

Procedure

  1. Go to the Talend Studio installation directory.
  2. Add the -Druntime.lineage=true attribute in the corresponding .ini file according to your operating system to enable the runtime lineage feature in Talend Studio.
  3. Save the file and start Talend Studio.
  4. Click Project settings on the toolbar of the Talend Studio main window or click File > Edit Project properties from the menu bar to open the Project Settings dialog box.
  5. In the tree view of the dialog box, expand the Job Settings node and then click Runtime lineage to display the corresponding view.
    Runtime Lineage configuration in the Project Settings dialog box.
  6. Enable runtime lineage for Standard Jobs via either of the following two ways:
    • To enable runtime lineage for all Standard Jobs, select the Use runtime lineage for all Jobs check box.
    • To enable runtime lineage for specific Standard Jobs, select the check boxes corresponding to the Jobs in the Use runtime lineage for selected Jobs area.
  7. In the Output path field, specify the path where you want to save the JSON files used by Talend Data Catalog.
    Later, each time you execute a Standard Job for which runtime lineage is enabled, a JSON file will be saved under a directory with the format <output_path>/<project>/<jobname>/<version>/runtime_log_<timestamp>.json, where
    • <output_path> is the path specified in the Output path field,
    • <project> is the name of the project,
    • <jobname> is the name of the Job,
    • <version> is the version of the Job, and
    • <timestamp> is the timestamp when the JSON file is generated.

    You can also set the output path by adding a JVM parameter -Druntime.lineage.outputpath=<output_path> for the Job via one of the following ways:

    Information noteNote: The output path must be specified for saving the JSON files. If the output path value is specified in multiple places, one of them will take effect according to the following precedence: 1) the value of the JVM parameter for specific Job, 2) the value of the Output path field, 3) the value of the JVM parameter for all Jobs, 4) the value of the JVM parameter in the shell script.
  8. Click Apply and Close to apply your changes and close the dialog box.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here