Talend Cloud Data Fabric Remote Engine Cluster - Batch Servers
All Environments
Remote Engine Clusters for Batch Servers provide the following functionality:
- Simplified deployment of artifacts - no need to deploy a Task individually when there is more than one server that could be used to execute it within an environment
- High Availability - ensuring that there is always a Remote Engine available to execute a task when scheduled or run on demand (assuming there is more than one server in the cluster)
- Load balancing - a round robin algorithm combined with information about a servers current resource usage is employed (assuming there is more than one server in the cluster)
- When combined with DevOps tools, the Talend scripts / templates from the IaaS marketplaces and the Talend Management Console APIs, simple auto-scaling can be achieved with Remote Engine Clusters
Remote Engines are a logical concept defined in the Talend Management Console and
require no additional infrastructure.
Development and Test
These environments show only a single Batch Server (that is, a single Remote Engine) in the cluster. Although high availability and load balancing are not required in these environments, the Remote Engine Cluster should still be created. This is due to the way that the Talend Management Console promotion pipeline process functions. Creating the Remote Engine cluster allows us to promote between environments as per our deployment strategy.
Pre-Production and Production
Automation should be leveraged to ensure any file that is used by a job is deployed to every server in the Remote Engine Cluster.