Known limitations of Dynamic Engine
- Dynamic Engine supports only data integration and big data Job tasks. Routes and data service Jobs are not supported.
- Each Kubernetes cluster can only have one Dynamic Engine by design. A Dynamic Engine provides the core infrastructure services for task and plan executions.
- There is no mechanism to prevent deploying more than one Dynamic Engine in a Kubernetes cluster. Doing this may cause unexpected connectivity and performance issues for Talend Management Console, Dynamic Engines, and Dynamic Engine environments.
- You must use the deployment files generated by Talend Management Console to deploy Dynamic Engines and Dynamic Engine environments. Custom deployment files are not supported.
- The Kubernetes cluster in which Dynamic Engine is deployed must have access to public internet.
- The following task execution features for data integration Jobs are not supported:
- Remote debugging of a Data Integration Job on Dynamic Engines
- User impersonation
- Custom signature check
- The tSystem component is not supported.
Local storage limitations in Dynamic Engine environments
You can use local file system components (such as tFileInputDelimited and tFileOutputDelimited) for temporary data access within a single Job. However, these cannot be used for persistent storage or inter-Job sharing due to the temporary nature of Kubernetes containers used by Dynamic Engine environments. Container data is destroyed after task completion.
As a result, cloud storage components like S3 or Azure are essential for data handling in Dynamic Engine environments. They connect to S3 or Azure storage solutions to transfer data between external storage and task execution containers, providing persistent storage and enabling data sharing between Jobs.
- Amazon S3 components
- Azure Data Lake Storage Gen2
- Azure Storage Blob
- Azure Storage Queue
- Azure Storage Table