Configuring Job size limits for Dynamic Engine environments

To protect your Dynamic Engine environments from Denial of Service (DoS) attacks, you can set limits on Job artifact sizes during deployment and execution. Any Job artifact that exceeds these limits is automatically deleted.

Customize Job size limits by creating a custom values file and applying it to your Dynamic Engine environment.

Before you begin

The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.
Your Dynamic Engine must have been deployed. If not, run this command to deploy it:
```
helm install dynamic-engine -f <engine-id>-helm-values/<engine-id>-values.yaml oci://ghcr.io/talend/helm/dynamic-engine 
```
This is the default deployment, because the current customization is only needed on the Dynamic Engine environment side. For this reason no customization is configured for the Dynamic Engine instance.

Procedure

Create a custom values file for your Dynamic Engine environment with the required Job size limits to protect your system from DoS attacks.

Example

cat <<EOF > <environment-id>-custom-job-limits-values.yaml
configuration:
  jobLimits: 
    maxZipNameLength: 240
    maxZippedEntries: 2048
    maxUnzippedSize: 1000000000
    maxUnzippedFolderNameLength: 240
    maxUnzippedFileNameLength: 240
    maxZipDepth: 64
EOF

Apply the custom values file to your Dynamic Engine environment using Helm.

helm install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml  \
 -f <environment-id>-custom-job-limits-values.yaml \
 oci://ghcr.io/talend/helm/dynamic-engine-environment \
 --version $DYNAMIC_ENGINE_VERSION

Tip: When changes are needed, you can update the content of the <environment-id>-custom-job-limits-values.yaml file and run helm upgrade to apply the changes:

helm upgrade dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml  \
-f <environment-id>-custom-job-limits-values.yaml \
oci://ghcr.io/talend/helm/dynamic-engine-environment \
--version $DYNAMIC_ENGINE_VERSION

Results

Verify that the jobLimits property appears in the ConfigMap called global-configuration for your Dynamic Engine environment:

"jobLimits": {
  "maxZipNameLength": 240,
  "maxZippedEntries": 2048,
  "maxUnzippedSize": 1000000000,
  "maxUnzippedFolderNameLength": 240,
  "maxUnzippedFileNameLength": 240,
  "maxZipDepth": 64
}

These fields specify the following size limits:

maxZipNameLength: Maximum length of the zip file name.
maxZippedEntries: Maximum number of files and directories in the zip file.
maxUnzippedSize: Maximum size of the uncompressed zip file in bytes.
maxUnzippedFolderNameLength: Maximum length of any folder name after unzipping.
maxUnzippedFileNameLength: Maximum length of any file name after unzipping.
maxZipDepth: Maximum nesting level of directories in each Job artifact zip file.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here