Using component observability metrics to build reports for task runs

Use POST at /monitoring/observability/metrics/component to build reports about task runs.

Procedure

Get the ID of the task you need to analyze.
The task ID is available on the Task details page in Talend Management Console. If you need to get this ID via API, follow Getting tasks for which you need update the artifact version.

Issue the following API request to find the historical runs of this task during a given period:

method: GET
endpoint: 
      https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions?from={start_timestamp_ms}&to={end_timestamp_ms}
headers: {
 "Authorization": "Bearer <personanl_access_token>"
}
payload: N/A

From the response, count the number of runs and find those that failed, if any.

Issue the following API request to find the execution duration of a specific type of component in each of the task runs:

method: POST
endpoint: 
      https://api.<env>.cloud.talend.com/monitoring/observability/metrics/component
headers: {
 "Authorization": "Bearer <personanl_access_token>"
}
payload: {
    "startTime": "job start time", 
    "endTime": "job end time", 
    "environmentId": "environment id value",
    "sortBy": "component_execution_duration_milliseconds",
    "sortOrder": "desc";
    "filters": [
        {
            "field": "task_id ",
            "operator": "in", 
            "value": ["ID_of_the_task_to_analyze"]
        },
        {
            "field": "connector_type ",
            "operator": "in", 
            "value": ["a_component_type_such_as_tMongoDBConnection"]
        }
    ]
}

Analyze the response to identify how the execution duration of this type of components varies from one task run to another.

Results

This report helps you understand the circumstances in which a component changes its behavior among task runs.

Below is an example of the response of the POST request at /monitoring/observability/metrics/component.

{
  "items": [
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tMongoDBConnection",
      "connector_label": "tMongoDBConnection_1",
      "connector_id": "tMongoDBConnection_1",
      "component_start_time_seconds": 1628266578
    },
    {
      "account_id": "2be59707-2230-45dc-a43d-db7e6d798425",
      "engine_id": "a60bb1c0-7669-407f-9326-138af05da18a",
      "engine_type": "CLOUD",
      "engine_version": "2.10.7",
      "workspace_id": "6093f6a36f01d639680d31cd",
      "task_id": "610d3b7da8abfa32ffb6506b",
      "task_execution_id": "2b252f47-ccfd-4c13-9c4a-d864b599086f",
      "operator": "xxx",
      "operator_type": "HUMAN",
      "process_id": "0329f8d4-1c69-3372-9233-d38ac6ef03a8",
      "job_name": "MainJob",
      "pid": "20210806181617_2Y68h",
      "father_pid": "20210806181617_2Y68h",
      "root_pid": "20210806181617_2Y68h",
      "connector_type": "tRowGenerator",
      "connector_label": "tRowGenerator_1",
      "connector_id": "tRowGenerator_1",
      "target_connector_type": "tFlowMeter",
      "target_label": "tFlowMeter_row1",
      "target_id": "tFlowMeter_row1",
      "connection_type": "row1",
      "connection_name": "output",
      "component_start_time_seconds": 1628266578,
      "component_connection_rows_total": 5000000,
      "component_execution_duration_milliseconds": 491585
    },
    ...
  ],
  "limit": 50,
  "offset": 0,
  "total": 7
}

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here