Known limitations of Dynamic Engine
- Each Kubernetes cluster can only have one Dynamic Engine by design. A Dynamic Engine provides the core infrastructure services for task and plan executions.
- There is no mechanism to prevent deploying more than one Dynamic Engine in a Kubernetes cluster. Doing this may cause unexpected connectivity and performance issues for Talend Management Console, Dynamic Engines, and Dynamic Engine environments.
- You must use the deployment files generated by Talend Management Console to deploy Dynamic Engines and Dynamic Engine environments. Custom deployment files are not supported.
- The Kubernetes cluster in which Dynamic Engine is deployed must have access to public internet.
- The following task execution features for data integration Jobs are not supported:
- Remote debugging of a Data Integration Job on Dynamic Engines
- User impersonation
- The tSystem component is not supported.
Local storage and persistent data in Dynamic Engine environments
Local file system components (such as tFileInputDelimited and tFileOutputDelimited) require persistent storage for data to survive across Job executions and be shared between Jobs. However, Kubernetes containers are ephemeral by default, and data is destroyed after task completion. You have two approaches to address this:
- Mount Kubernetes resources:
You can mount Kubernetes resources in your task containers and configure local file components to read from or write to the mounted path. This provides persistent storage in your Dynamic Engine environment without relying on external cloud services.
For details, see Mounting custom Kubernetes resources in Dynamic Engine environments.
- Use cloud storage components (S3, Azure, or equivalent):
Cloud storage components (S3, Azure Data Lake Storage, Azure Blob, etc.) connect Dynamic Engine to external storage solutions, providing persistent data handling beyond the cluster.
For further information about how to use cloud storage components in your Jobs, see: