Configuring autoscaling for Data Services and Routes

Configure Horizontal Pod Autoscaling (HPA) for the resource allocated to Data Services and Routes in a Dynamic Engine environment, ensuring that Data Services and Routes are capable of handling large volumes of traffic or messages.

Note: This procedure applies to deployments and upgrades of a Dynamic Engine environment. You can change these settings with a subsequent upgrade.

Before you begin

The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd Helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.
Your Dynamic Engine must have been deployed. If not, run this command to deploy it:
```
helm install dynamic-engine -f <engine-id>-helm-values/<engine-id>-values.yaml oci://ghcr.io/talend/helm/dynamic-engine 
```
This is the default deployment, because the current customization is only needed on the Dynamic Engine environment side. For this reason no customization is configured for the Dynamic Engine instance.
The support for autoscaling relies on a Metrics server.

To verify if a Metrics server is installed, run:
```
kubectl top pod
```
If you get an error such as error: Metrics API not available, you need to install a Metrics server.

About this task

To activate Pod autoscaling for Data Services and Routes, configure the dynamic-engine-environment chart. This configuration also allows setting the maximum number of replicas (Pod instances) that the HPA can scale up to.

Procedure

In the Kubernetes machine, unzip the Helm deployment zip file previously downloaded.

Create a custom autoscaling-specific values file for Data Services and Routes.

Example

cat <<EOF > $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-autoscaling-ds-routes.yaml
configuration:
  dataServiceRouteDeployment:
    autoscaling:
      enabled: true
      maxReplicas: 5
EOF

Replace $DYNAMIC_ENGINE_ENVIRONMENT_ID with the ID of your Dynamic Engine environment. In this example, the upper limit of Pod autoscaling is set to 5 Pod instances.

List of parameters that can be set at autoscaling level:

Autoscaling parameters for Data Services and Routes
Parameter	Description	Default
enabled	Specifies whether a Horizontal Pod Autoscaler (HPA) is automatically deployed with the environment.	false
minReplicas	Sets the minimum number of pod replicas for autoscaling.	1
maxReplicas	Sets the maximum number of pod replicas for autoscaling.	10
cpuAverageUtilization	Sets the target average CPU utilization (percentage) across pods. Use a negative value to deactivate this parameter. Zero uses the default value (70).	0
memoryAverageUtilization	Sets the target average memory utilization (percentage) across pods. Use a negative value to deactivate this parameter. Zero uses the default value (-1).	0
scaleUpStabilizationWindowSeconds	Sets the stabilization window (in seconds) for scaling up. During this window, previously calculated target replica counts are considered. Zero deactivates this feature.	120
scaleDownStabilizationWindowSeconds	Sets the stabilization window (in seconds) for scaling down. During this window, previously calculated target replica counts are considered. Zero deactivates this feature.	0

Deploy the Dynamic Engine environment with the custom autoscaling-specific values file.

helm install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
   -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml  \
   -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-autoscaling-ds-routes.yaml \
   oci://ghcr.io/talend/helm/dynamic-engine-environment \
   --version $DYNAMIC_ENGINE_VERSION

Replace $DYNAMIC_ENGINE_ENVIRONMENT_ID and $DYNAMIC_ENGINE_VERSION with the respective environment ID and engine version.

Verify the autoscaling configuration in the ConfigMap of the environment.

Each Dynamic Engine environment has its own dedicated global-configuration ConfigMap. Updates to a specific ConfigMap only affect the environment to which it belongs.

kubectl get configmap -n qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
  global-configuration -ojsonpath="{.data['global-configuration\.json']}" \
  | jq -r '.dataServiceRouteDeployment'

This command retrieves the Pod autoscaling configuration. The output should confirm the enabled status and the specified maximum replicas:

{
  "autoscaling": {
    "enabled": "true",
    "minReplicas": "1",
    "maxReplicas": "5",
    "cpuAverageUtilization": "0",
    "memoryAverageUtilization": "0"
  }
}

Tip: When changes are needed, you can update the content of the $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-autoscaling-ds-routes.yaml file and run helm upgrade to apply the changes:

helm upgrade dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
   -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml  \
   -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-autoscaling-ds-routes.yaml \
   oci://ghcr.io/talend/helm/dynamic-engine-environment \
   --version $DYNAMIC_ENGINE_VERSION

Results

After successful deployment, the Dynamic Engine environment can automatically increase Pod instances according to the workload variations when running Data Services and Routes, up to the configured maximum.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

What to do next

The autoscaling settings configured at the environment level apply to all Data Service and Route tasks running on that environment by default. However, you can override these environment-level settings using run profile-level autoscaling parameters. This allows you to customize scaling behavior for specific tasks through the run profile-level autoscaling parameters, without changing environment-wide settings.

Run profile-level autoscaling parameters are notably useful when:

A particular Data Services or Routes task has different performance characteristics and requires different scaling thresholds than the environment default.
You want to limit scaling for resource-intensive tasks to prevent cluster overload.
A task is critical and requires minimum replicas higher than the environment default.

To override environment-level autoscaling for a task, create or update a run profile with the desired autoscaling parameters as JVM arguments (for example: -K8S:AutoscalingMaxReplicas=5). Any task that uses that run profile will then inherit these customized settings, which override the corresponding environment-level configuration.

Note: Run profile-level autoscaling parameters are available in Dynamic Engine version 1.2.0 and later.

For a complete list of autoscaling parameters and their corresponding run profile arguments, see Configuring autoscaling parameters at the task level.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here