Prerequisites to use Dynamic Engine

Access permissions to Talend Management Console and Dynamic Engine

• You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role to access the Dynamic Engine tab in Talend Management Console.
• You must have the Environment Administrator (contains the TMC_ENVIRONMENT_MANAGEMENT permission) role to access the Dynamic Engine Environments tab.

Kubernetes version compatibility

The following table lists the Kubernetes versions supported by Dynamic Engine:

Supported operating systems and architectures

The Kubernetes worker nodes must run on a Linux operating system with one of the following supported CPU architectures:

• Linux x86-64 (linux/amd64)
• Linux ARM64 (linux/arm64)

Windows Server is not supported. Only Linux systems are compatible.

If you do not have a Kubernetes cluster yet, create one either locally or on a cloud platform such as AWS, Google Cloud, or Azure. Although not officially supported, other public cloud providers such as Digital Ocean or OpenShift should work as well, provided they use standard Kubernetes distributions.

Cluster administrator permissions and minimum requirements

You must be the administrator of the Kubernetes cluster to deploy and manage Dynamic Engine in this cluster.

Your Kubernetes cluster must meet the following minimum requirements:

• Instance size: 16 GB of memory and 4 vCPU
• Number of nodes: 3
• Disk size: 20 GB (including 2 GB minimum for running the engine plus space for deployed Job artifacts)
• Persistent storage with CSI (Container Storage Interface) provider is required. The storage must be accessible from any node (the Kubernetes ReadWriteMany mode). See the following Supported cloud platforms and storage providers section for details.

Supported cloud platforms and storage providers

The following table provides a quick reference for supported cloud platforms, their Kubernetes distributions, and corresponding storage providers:

For details specific to each storage provider, see the following:

• Amazon EFS: See Amazon EFS CSI Driver for mounting and configuration instructions.
• Azure Files: See Create and use a volume with Azure Files in Azure Kubernetes Service (AKS) for configuration details.
• Filestore: See Filestore CSI Driver for further information. Dynamic Engine v1.0.0 and later are officially supported on GKE. For an example of deploying a Dynamic Engine instance and its environment to GKE, see Deploying Dynamic Engine in a GKE cluster.

For local development and testing on a minikube, you can use Local Path Provisioner.

Persistent storage configuration for Dynamic Engine services

As described in the table above, persistent storage is required for Dynamic Engine services to operate. In Kubernetes, persistent storage is provided through storage classes, which define how and where disk space is dynamically allocated when a service requests it.

When configuring the storage class for any CSI storage provider, you can use either the default Kubernetes storage class or a storage class dedicated to Dynamic Engine:

• Using the default storage class: Add the following annotation to set it as the cluster default:

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"

  This annotation ensures that when a persistent volume is requested without specifying a storage class, Kubernetes automatically uses this default storage class.

• Using a dedicated storage class for Dynamic Engine: See Provisioning a storage class dedicated to Dynamic Engine environment services.

Requirements for advanced features

Metrics server: If you plan to configure autoscaling for Data Services and Routes tasks, you must have a Metrics server installed in your Kubernetes cluster.

To verify if a Metrics server is already installed, run:

kubectl top pod

If you get an error such as error: Metrics API not available, install a Metrics server before configuring autoscaling.

For details on configuring autoscaling, see Configuring autoscaling for Data Services and Routes.

DNS URLs to be added in your proxy or firewall allowlist

For detailed URL list specific to your region, see Adding URLs to your proxy and firewall allowlist.