Skip to main content Skip to complementary content

Monitoring Job runs on Remote Engines

You can observe your Data Integration Jobs running on Talend Remote Engines.

Information noteNote: This feature requires Talend Studio version 7.3 R2020-07 or later versions.

If your Jobs are scheduled to run on Talend Remote Engine version 2.9.2, Talend Management Console can provide the data needed to build your own customized dashboards in your company monitoring system or an Application Performance Monitoring (APM) tool such as ELK, Datadog, Azure Monitor, or Amazon CloudWatch.

Once you have set up the metric and log collection system in Talend Remote Engine and your APM, you can design and organize your dashboards thanks to the information sent from Talend Management Console to the APM through the engine.

For example, with this data you can monitor in your dashboards:
  • Job run statuses
  • Talend Remote Engine availability
  • Resource consumption (CPU, memory, disk usage)
  • Workload (number of tasks running)
Thanks to the monitoring capabilities, you can also set up alerts in case errors occurs, and optimize Job run performance by comparing its duration and the records processed.

Available metrics

The following metrics and additional logs are provided by Talend Remote Engine that you can integrate in your APM tool.

Already existing logs from previous Remote Engine versions:
  • Job run logs
  • Remote Engine logs
New logs from Talend Remote Engine v2.9.2 or later:
  • Job management logs
  • Performance metrics and context of Jobs, Remote Engines, and operating system
    • OS resource consumption
    • Remote Engine resource consumption and workload
    • Job run resource consumption (CPU, memory)
    • Job run history (including component information, status, and duration)
    • Job run performance (for example, records processed or rejected)
The metrics are available in two different formats, accessible from an HTTP endpoint exposed by the Remote Engine:
  • generic JSON file
  • Prometheus-compatible format

Using the feature

To start using the monitoring feature, you must:
  1. Download and install the new Remote Engine version.
  2. Set up the Remote Engine on your development environment in Talend Management Console.
  3. Install/update Talend Studio to version 7.3 R2020-07 or later.
  4. Republish your Jobs from the new Talend Studio version to Talend Management Console.
  5. Update you tasks to use the latest Jobs versions (if not done automatically).
  6. Run your Jobs.

Setting up a connection between the APM and the Talend apps depends on which APM you use. You can access How to configure observability metrics with Talend Cloud - Remote Engine examples on Qlik Community.

By default, you can view the metrics in the Run overview page of a task in Talend Management Console. If you need to prevent Talend Remote Engine from sending metrics to Talend Management Console, contact the Support team.

In this section

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here