Skip to main content Skip to complementary content

Using notifications and logs to debug a pipeline

How to use notifications and logs to spot and correct errors in a pipeline which execution has failed.

Before you begin

  • You have previously created a complete pipeline.

  • You have executed your pipeline and the execution failed.

Procedure

  1. Open the pipeline that failed.
    A pipeline shows a customer table dataset as the pipeline source, a Python 3 processor, and an HDFS dataset as the pipeline destination.

    Here, this pipeline contains a database table about customers, a Python 3 processor that concatenate names, converts currency and processes dates, and a HDFS topic to store the processed data.

  2. To display the error message that informs you about the execution status, click the Notification icon on the top toolbar.
    The notification center indicates that the pipeline started, then ended on an error.

    From that window, you can expand the error message by clicking on it.

  3. To get more detailed and categorized information about your error, either click the View Logs link from the Notification window, or go to the Metrics tab of the Pipeline Details panel and click the View Logs button.

    By default, all logs are displayed.

  4. To sort the logs and only read the error messages, select the Error check box.
    In the Logs panel, the error check box is selected and an error message related to an incorrect operator on line 7 is highlighted.

    You can see that the error is linked to an incorrect operator used in the Python 3 processor on line 7.

  5. Now that you have spotted the error, select the Python 3 processor to edit its configuration.
    The Python 3 code editor is open and the line 7 is highlighted.

    On line 7 here, the year 2017 is surrounded by unnecessary operators that cause the pipeline to fail. Edit the line as follows:output['number_year_registrated'] = 2017 - int(year) and save your changes.

  6. Click the run icon on the top toolbar to execute your pipeline again.

Results

The pipeline is fixed and is executed with success. If you missed the notification informing about the execution success, click the Notification icon on the top toolbar to display it.
The notification center indicates that the pipeline started, then ended successfully.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here