Prerequisites to use Dynamic Engine
Access permissions to Talend Management Console and Dynamic Engine
- You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role to access the Dynamic Engine tab in Talend Management Console.
- You must have the Environment Administrator (contains the TMC_ENVIRONMENT_MANAGEMENT permission) role to access the Dynamic Engine Environments tab.
Kubernetes specifications
- If you do not have any Kubernetes cluster yet, create one either locally or on a Cloud platform, such as AWS or Azure.
- The Kubernetes cluster must be one of the versions from v1.27 to v1.30.
- You must be the administrator of the Kubernetes cluster. This ensures that you have enough rights and permissions to deploy and manage Dynamic Engine in this cluster.
- Minimum requirements for the Kubernetes cluster:
- Instance size: 16 GB of memory and 4 vCPU
- Number of nodes: 3
- Disk size: 20 GB (including 2 GB minimum for running the engine plus space for
deployed Job artifacts)
This requires the availability of CSI provider and a corresponding storage class configured. This CSI provider and storage class allow your Kubernetes cluster to have persistent storage that is accessible from any node.
The supported CSI storage providers are
- Amazon EFS. For further information about mounting this persistent storage, see Amazon EFS CSI Driver.
- Azure Files. For further information, see Create and use a volume with Azure Files in Azure Kubernetes Service (AKS).
- If you cannot use either of the above providers, install Longhorn, a distributed block storage system for Kubernetes, in your cluster. With Longhorn, the required minimum disk size is 100 GB. For further information about installing Longhorn, see (Optional) Setting up a Longhorn storage solution for your Kubernetes cluster.
Note that for any CSI storage providers, when configuring the storage class, the following annotation must be added:This annotation ensures that the configured storage class is set as the default in the Kubernetes cluster. As a result, when a persistent volume is requested without specifying a storage class, Kubernetes will automatically use this default storage class.apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true"
Information noteTip: Although not officially supported, other public cloud providers, such as Google Cloud, Digital Ocean, or Open Shift, should work as well as the supported ones, provided these providers use standard Kubernetes distributions.
DNS URLs to be added in your proxy or firewall allowlist
Region: AWS APAC data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://api.ap.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required https://pair.ap.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.ap.cloud.talend.com or wss://engine.ap.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.ap.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
Dynamic Engine Provide Helm charts Required Region: AWS Australia data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://api.au.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required https://pair.au.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.au.cloud.talend.com or wss://engine.au.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.au.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
Dynamic Engine Provide Helm charts Required Region: AWS EU data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://api.eu.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required https://pair.eu.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.eu.cloud.talend.com or wss://engine.eu.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.eu.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
Dynamic Engine Provide Helm charts Required Region: AWS US data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://api.us.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required https://pair.us.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.us.cloud.talend.com or wss://engine.us.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.us.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
Dynamic Engine Provide Helm charts Required - Region: Azure US West data center
The port of all these URLs is 443. Their traffic is outbound.
URL Used by Purpose Requirement level https://api.us-west.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required https://pair.us-west.cloud.talend.com Dynamic Engine Send the initial pairing request, and heartbeats
Required https://engine.us-west.cloud.talend.com
or
wss://engine.us-west.cloud.talend.com
Dynamic Engine Pipeline task executions Required https://vault-gateway.us-west.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required https://msg.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-artifacts.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required https://minio-customerlogs.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required ghcr.io
and
docker.io
Dynamic Engine Provide Docker images Required *.talend.github.io
Dynamic Engine Provide Helm charts Required