Configuring autoscaling for Data Services and Routes

Configure Horizontal Pod Autoscaling (HPA) for Data Services and Routes in a Dynamic Engine environment.

Note: This procedure only applies to deployments and upgrades of a Dynamic Engine environment. You can change these settings with a subsequent upgrade.

About this task

To activate Pod autoscaling for Data Services and Routes, configure the dynamic-engine-environment chart. This configuration also allows setting the maximum number of replicas (Pod instances) that the HPA can scale up to.

Procedure

Use the dynamic-engine-crd Helm chart to install the Dynamic Engine custom resource definitions of a given Helm version.
1. The chart version to use is the same as your Dynamic Engine version. You can find this version using one of the following methods:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. In the Kubernetes cluster, run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.
Still in the Kubernetes machine, unzip the Helm deployment zip file previously downloaded.
Create a custom autoscaling-specific values file for Data Services and Routes.
Example
```
cat <<EOF > <environment-id>-custom-autoscaling-ds-routes.yaml
configuration:
  dataServiceRouteDeployment:
    autoscaling:
      enabled: true
      maxReplicas: 5
EOF
```
Replace <environment-id> with the ID of your Dynamic Engine environment. In this example, the upper limit of Pod autoscaling is set to 5 Pod instances.
Run this command to deploy the Dynamic Engine instance:
```
helm install dynamic-engine -f <engine-id>-helm-values/<engine-id>-values.yaml oci://ghcr.io/talend/helm/dynamic-engine
```
This is the default deployment, because the autoscaling customization is only needed on the Dynamic Engine environment side. So no customization has been done for the Dynamic Engine instance.

Deploy the Dynamic Engine environment with the custom autoscaling-specific values file.

helm install dyn-engine-env-<environment-id> -f <environment-id>-values.yaml  \
-f <environment-id>-custom-autoscaling-ds-routes.yaml \
oci://ghcr.io/talend/helm/dynamic-engine-environment \
--version <engine-version>

Replace <environment-id> and <engine-version> with the respective environment ID and engine version.

Verify the autoscaling configuration in the ConfigMap of the environment.
Each Dynamic Engine environment has its own dedicated global-configuration ConfigMap. Updates to a specific ConfigMap only affect the environment to which it belongs.
```
kubectl get configmap -n qlik-processing-env-$DYN_ENGINE_ENV_ID global-configuration -ojsonpath="{.data['global-configuration\.json']}"
 | jq -r '.dataServiceRouteDeployment'               
```
This command retrieves the Pod autoscaling configuration. The output should confirm the enabled status and the specified maximum replicas:
```
{
  "autoscaling": {
    "enabled": "true",
    "minReplicas": "1",
    "maxReplicas": "5",
    "cpuAverageUtilization": "0",
    "memoryAverageUtilization": "0"
  }
}         
```
Information noteTip: When changes are needed, you can update the content of the <environment-id>-custom-autoscaling-ds-routes.yaml file and run helm upgrade to apply the changes:
```
helm upgrade dyn-engine-env-<environment-id> -f <environment-id>-values.yaml  \
-f <environment-id>-custom-autoscaling-ds-routes.yaml \
oci://ghcr.io/talend/helm/dynamic-engine-environment \
--version <engine-version>      
```

Results

After successful deployment, the Dynamic Engine environment can automatically increase Pod instances according to the workload variations when running Data Services and Routes, up to the configured maximum.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here