Skip to main content Skip to complementary content

Prerequisites to use Dynamic Engine

The requirements to use Dynamic Engine include DNS URLs to be added to your proxy and firewall allowlist, Kubernetes specifications, and access permissions to Talend Management Console and Dynamic Engine.

Access permissions to Talend Management Console and Dynamic Engine

  • You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role to access the Dynamic Engine tab in Talend Management Console.
  • You must have the Environment Administrator (contains the TMC_ENVIRONMENT_MANAGEMENT permission) role to access the Dynamic Engine Environments tab.
For further information about roles and their respective permissions in Talend Management Console, see Pre-defined user roles.

Kubernetes specifications

  • If you do not have any Kubernetes cluster yet, create one either locally or on a Cloud platform, such as AWS or Azure.
    • The Kubernetes worker nodes must run on a linux_x86_64 architecture, an x86-64 (64-bit) CPU architecture running on a Linux operating system. This requirement means:
      • ARM-based processor architecture is not supported.
      • Windows Server is not supported. Only Linux system is compatible.
  • The Kubernetes cluster must be one of the versions from v1.27 to v1.30.
  • You must be the administrator of the Kubernetes cluster. This ensures that you have enough rights and permissions to deploy and manage Dynamic Engine in this cluster.
  • Minimum requirements for the Kubernetes cluster:
    • Instance size: 16 GB of memory and 4 vCPU
    • Number of nodes: 3
    • Disk size: 20 GB (including 2 GB minimum for running the engine plus space for deployed Job artifacts)

      This requires the availability of CSI provider and a corresponding storage class configured. This CSI provider and storage class allow your Kubernetes cluster to have persistent storage that is accessible from any node.

      The supported CSI storage providers are

      Note that for any CSI storage providers, when configuring the storage class, the following annotation must be added:
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
         annotations:
          storageclass.kubernetes.io/is-default-class: "true"
      This annotation ensures that the configured storage class is set as the default in the Kubernetes cluster. As a result, when a persistent volume is requested without specifying a storage class, Kubernetes will automatically use this default storage class.
    Information noteTip: Although not officially supported, other public cloud providers, such as Google Cloud, Digital Ocean, or Open Shift, should work as well as the supported ones, provided these providers use standard Kubernetes distributions.

DNS URLs to be added in your proxy or firewall allowlist

  • Region: AWS APAC data center

    The port of all these URLs is 443. Their traffic is outbound.

    URL Used by Purpose Requirement level
    https://api.ap.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required
    https://pair.ap.cloud.talend.com Dynamic Engine

    Send the initial pairing request, and heartbeats

    Required

    https://engine.ap.cloud.talend.com or wss://engine.ap.cloud.talend.com

    Dynamic Engine Pipeline task executions Required
    https://vault-gateway.ap.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required
    https://msg.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-artifacts.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-customerlogs.ap.cloud.talend.com Dynamic Engine Data Integration Job task executions Required

    ghcr.io

    and

    docker.io

    Dynamic Engine Provide Docker images Required

    *.talend.github.io

    Dynamic Engine Provide Helm charts Required
  • Region: AWS Australia data center

    The port of all these URLs is 443. Their traffic is outbound.

    URL Used by Purpose Requirement level
    https://api.au.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required
    https://pair.au.cloud.talend.com Dynamic Engine

    Send the initial pairing request, and heartbeats

    Required

    https://engine.au.cloud.talend.com or wss://engine.au.cloud.talend.com

    Dynamic Engine Pipeline task executions Required
    https://vault-gateway.au.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required
    https://msg.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-artifacts.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-customerlogs.au.cloud.talend.com Dynamic Engine Data Integration Job task executions Required

    ghcr.io

    and

    docker.io

    Dynamic Engine Provide Docker images Required

    *.talend.github.io

    Dynamic Engine Provide Helm charts Required
  • Region: AWS EU data center

    The port of all these URLs is 443. Their traffic is outbound.

    URL Used by Purpose Requirement level
    https://api.eu.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required
    https://pair.eu.cloud.talend.com Dynamic Engine

    Send the initial pairing request, and heartbeats

    Required

    https://engine.eu.cloud.talend.com or wss://engine.eu.cloud.talend.com

    Dynamic Engine Pipeline task executions Required
    https://vault-gateway.eu.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required
    https://msg.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-artifacts.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-customerlogs.eu.cloud.talend.com Dynamic Engine Data Integration Job task executions Required

    ghcr.io

    and

    docker.io

    Dynamic Engine Provide Docker images Required

    *.talend.github.io

    Dynamic Engine Provide Helm charts Required
  • Region: AWS US data center

    The port of all these URLs is 443. Their traffic is outbound.

    URL Used by Purpose Requirement level
    https://api.us.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required
    https://pair.us.cloud.talend.com Dynamic Engine

    Send the initial pairing request, and heartbeats

    Required

    https://engine.us.cloud.talend.com or wss://engine.us.cloud.talend.com

    Dynamic Engine Pipeline task executions Required
    https://vault-gateway.us.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required
    https://msg.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-artifacts.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-customerlogs.us.cloud.talend.com Dynamic Engine Data Integration Job task executions Required

    ghcr.io

    and

    docker.io

    Dynamic Engine Provide Docker images Required

    *.talend.github.io

    Dynamic Engine Provide Helm charts Required
  • Availability-noteBeta
    Region: Azure US West data center

    The port of all these URLs is 443. Their traffic is outbound.

    URL Used by Purpose Requirement level
    https://api.us-west.cloud.talend.com/processing/runtimes/dynamic-engines Dynamic Engine Connectivity status Required
    https://pair.us-west.cloud.talend.com Dynamic Engine

    Send the initial pairing request, and heartbeats

    Required

    https://engine.us-west.cloud.talend.com

    or

    wss://engine.us-west.cloud.talend.com

    Dynamic Engine Pipeline task executions Required
    https://vault-gateway.us-west.cloud.talend.com Dynamic Engine Decrypt secrets used in pipeline task executions Required
    https://msg.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-artifacts.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required
    https://minio-customerlogs.us-west.cloud.talend.com Dynamic Engine Data Integration Job task executions Required

    ghcr.io

    and

    docker.io

    Dynamic Engine Provide Docker images Required

    *.talend.github.io

    Dynamic Engine Provide Helm charts Required

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!