Execution engines in Talend Cloud
Cloud Engine
- A default Cloud Engine is automatically available. It
cannot be assigned to any specific environment, that is to say, this
default Cloud Engine is always an unassigned
engine.
This engine is therefore not displayed in the Environment details blade.
- Unassigned Cloud Engines can be used by all environments. If Cloud Engines
are not allocated to environments, you may not be able to run certain tasks,
because other tasks keep all the engines occupied.
The default environment can use unassigned Cloud Engines only.
- Each Cloud Engine, including this default Cloud Engine,
requires 45000 engine tokens to be able to run and allows up to 3 tasks to run in
parallel on it. These 45000 tokens are released once this engine is shut down.
If you choose to run your task on Cloud Exclusive, no other task will be executed on that Cloud Engine. Cloud Exclusive engines can only be used in environments that do not have Cloud Engines assigned to them.
If three different tasks or the same task is already running on a Cloud Engine, another engine will be selected to the task execution.
The following system resource limitations are valid for Jobs running on Cloud Engines:
- vCPU: 2
- Disk usage: 200GB
- Memory usage: 8GB
- Throughput: 225GB per hour
For Jobs that require more resources, consider using Talend Remote Engine as the target runtime.
- A running Cloud Engine is verified every one hour with regard to its usage.
If no task run has been found during this hour, this Cloud Engine is
automatically shut down; otherwise, it keeps running and another shutdown
verification is scheduled for the end of another one hour. This is the
general lifecycle of a Cloud Engine.
However, if a warm-up event occurs, for example, a task run schedule is set up on this Cloud Engine, the ongoing shutdown schedule is canceled and the shutdown verification is re-scheduled for the end of another hour.
If your license is the community level, user login to Talend Cloud is also considered to be a warm-up event. In this case, the default Cloud Engine is started upon user login.
This shutdown verification mechanism helps avoid shutting down a Cloud Engine too frequently, as starting a Cloud Engine always requires quite some ramp-up time.
Dynamic Engine
Dynamic Engine is a cloud-native execution platform designed for scalability and advanced data processing, leveraging Kubernetes for resource orchestration and scaling. It is managed through Talend Management Console, which simplifies version upgrades and provides built-in notifications for monitoring.
Each Dynamic Engine environment requires 18,000 engine tokens to operate.
Currently, Dynamic Engine supports AWS EKS, Azure AKS, and on-premises Kubernetes clusters, with plans to expand support for additional platforms. For detailed setup and management instructions, see About Talend Management Console and Dynamic Engine.
Talend Remote Engine
Remote Engines allow you to run Job, Route, and data service tasks that use on-premises applications and databases. By default, maximum 3 tasks can run in parallel on the same engine. If you are using a Remote Engine to run identical tasks simultaneously, your Remote Engine must be v2.12.0 onwards.
Outbound communication between Talend Management Console and Talend Remote Engine is fully secured as data is not staged.
Data service and Route tasks can only be deployed on Remote Engines. OSGi type deployments require that Talend Runtime 7.1.1 version or later is installed and running on the same machine as the Talend Remote Engine. Microservices do not require any other application for Route and REST Data service deployments, except the Data Service Runner module inside the Remote Engine. Data service tasks can be executed only on engines that have a Microservice or Talend Runtime (for OSGi) run profile configured.
Remote Engines can be grouped together to form clusters. Any number of clusters can be created in an environment. However, Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.
A Remote Engine requires 9000 engine tokens to be able to run.
For more information about Talend Remote Engine and run profiles linked to Talend Studio artifacts, see Configuring Job run profiles.
Cloud Engine for Design
The Cloud Engine for Design is a built-in runner that allow you to easily design pipelines without having to set up any processing engines. You can execute pipelines with different resource allocations on a Cloud Engine for Design.
The following resource limitations are valid for pipelines running on Cloud Engine for Design:
- Memory usage: 8GB
- Two pipelines can be run in parallel
For advanced processing of data it is recommended to install the secure Remote Engine Gen2.
Remote Engine Gen2
A Remote Engine Gen2 is a secure execution engine on which you can safely run pipelines. By using this engine, you can make sure that data processing happens in a safe and secure environment as Talend never has access to the data and resources of your pipelines. You have control over your execution environment and resources as you are able to create and configure the engine in your own environment (Virtual Private Cloud or on premises). The engine also ensures optimal performance and security by increasing the data locality instead of moving large data to computation.
You can execute pipelines with different resource allocations on a Remote Engine Gen2 by using run profiles.
For more information about Remote Engine Gen2, see the Talend Remote Engine Gen2 Quick Start Guide.