Managing Run profiles for Dynamic Engine environment tasks

Run profiles contain the set of configurations, that is the resource allocation and advanced properties, for executing tasks.

You can see the list of run profiles from the Run Profiles tab of the Processing page. The list includes a Processor type column that identifies whether each run profile uses a Dynamic Engine processor.

You can use the Environment filter to display the list of run profiles associated with a specific Dynamic Engine environment.

You must have the Run Profiles - Manage (TMC_RUN_PROFILE_MANAGEMENT) user permission to manage run profiles.

Creating a run profile

Add a new run profile to configure JVM arguments, Kubernetes pod resources, and autoscaling parameters for Dynamic Engine tasks.

Before you begin

You must have the Run Profiles - Manage (TMC_RUN_PROFILE_MANAGEMENT) user permission to create run profiles.

Procedure

Open the Processing page.
Click the Run profiles tab.
Click Create new then Run profile.
Select Dynamic Engine as processor type.
Select a Dynamic Engine environment from the list.
Enter a name for the run profile.
Add a description.
Configure the Processing arguments with the JVM arguments you want to set for all Jobs running on the engine with this profile.
The JVM parameter values set here will override the values set in Talend Studio. The resource settings for Kubernetes pods and containers are also set here through the processing arguments.
This example shows custom settings for CPU and memory allocated for the current task run in Kubernetes. For further explanations about these parameters, see Resource settings for Talend Management Console task executions in this guide or Resource Management for Pods and Containers from the Kubernetes documentation.

Example
Processing argument groups for Dynamic Engine run profiles
```
-K8S:PodCPURequest=500m
-K8S:PodCPULimit=1
-K8S:PodRAMRequest=512M
-K8S:PodRAMLimit=1G
```
- -K8S:PodCPURequest specifies the initial CPU allocation for the pod.
- -K8S:PodCPULimit sets an upper limit on the pod's CPU usage, preventing it from monopolizing CPU resources.
  Information noteNote: Kubernetes measures CPU resources in milliCPU (m), also known as millicore. The smallest value you can specify is 1m, which represents 0.001 of a physical or virtual CPU core. If you enter a numeric value without a unit, Kubernetes interprets it as a fraction of a CPU core. For example:
  
  0.5 requests half a CPU core (500m).
  
  1 requests one full CPU core (1000m).
- -K8S:PodRAMRequest specifies the initial memory allocation for the pod.
- -K8S:PodRAMLimit sets an upper limit on the pod's memory usage, preventing it from consuming excessive resources.

For Data Service and Route tasks, you can override the environment-level autoscaling settings by adding run profile-level autoscaling parameters.

The Dynamic Engine environment may have autoscaling settings that apply to all Data Service and Route tasks. You can override these environment-level settings by adding autoscaling parameters to the Processing arguments section of a specific run profile. Tasks that use an engine with that run profile will then inherit these overridden settings, allowing fine-grained control over how Dynamic Engine instances scale in response to workload variations for specific tasks.

For more information about environment-level autoscaling configuration, see Configuring autoscaling for Data Services and Routes.

Example

Note: Run profile-level autoscaling parameters are available in Dynamic Engine version 1.2.0 and later.

Run profile-level autoscaling parameters
Run profile parameter	Description
-K8S:Replicas	Sets the number of pod replicas. When autoscaling is deactivated, this is the fixed replica count. When autoscaling is activated, this is the initial replica count. Default: 1
-K8S:AutoscalingEnabled	Activates or deactivates autoscaling for this task. Default: false Information noteImportant: When autoscaling is activated for a task, you cannot deactivate it when the task is still running, neither through configuration nor with an automatic re-deployment triggered by a task upgrade. To deactivate autoscaling, you must undeploy and then redeploy the task.
-K8S:AutoscalingMinReplicas	Sets the minimum number of pod replicas for autoscaling. Default: 1
-K8S:AutoscalingMaxReplicas	Sets the maximum number of pod replicas for autoscaling. Default: 10
-K8S:AutoscalingCpuAverageUtilization	Sets the target average CPU utilization (percentage) across pods. Use a negative value to deactivate this parameter; zero means using 70%. Default: 0
-K8S:AutoscalingMemoryAverageUtilization	Sets the target average memory utilization (percentage) across pods. Use zero or a negative value to deactivate this parameter. Default: 0
-K8S:AutoscalingScaleUpStabilizationWindowSeconds	Sets the stabilization window (in seconds) for scaling up. During this window, previously calculated target replica counts are considered. Zero deactivates this feature. Default: 120
-K8S:AutoscalingScaleDownStabilizationWindowSeconds	Sets the stabilization window (in seconds) for scaling down. During this window, previously calculated target replica counts are considered. Zero deactivates this feature. Default: 0

For big data Jobs, ensure you set -K8S:ImageProfile=standard when the environment-level global.profile configuration is minimal. Big data Jobs do not support minimal images and must use standard base images.

Note: The -K8S:ImageProfile run profile parameter is available in Dynamic Engine version 1.6.0 and later.

For more information on setting up distroless images for Dynamic Engine, see Deploying Dynamic Engine with minimal images.

Run profile-level image profile parameter
Run profile parameter	Description
-K8S:ImageProfile	Overrides the environment-level global.profile Helm value for the individual task. Use this parameter to run a specific task with a different image profile than the engine default. Accepted values: minimal or standard Default: inherits engine global.profile setting. Information noteImportant: Set -K8S:ImageProfile=standard for each big data task when the engine is deployed with global.profile: minimal. Big data Jobs are incompatible with minimal images and must use standard base images.

Click Save or Save and add another to create another run profile with different JVM settings, pod limits, or autoscaling parameters.

Updating a run profile

Modify an existing run profile to change JVM arguments, resource limits, autoscaling parameters, or image profile settings.

Before you begin

You must have the Run Profiles - Manage (TMC_RUN_PROFILE_MANAGEMENT) user permission to update run profiles.

Procedure

Open the Processing page.
Click the Run profiles tab.
Right-click the name of the run profile you want to edit then click Edit.
You can also click the name of the run profile to open the run profile details panel, then click Edit to open the drawer.
Edit the run profile, such as renaming the run profile, editing the description or editing the Processing arguments.
Example

Processing argument groups for Dynamic Engine run profiles
If this run profile will be used for big data Jobs and the engine is deployed with global.profile: minimal, ensure you set -K8S:ImageProfile=standard in the Processing arguments.

Information noteImportant: Big data Jobs (Spark Batch and Spark Streaming) are incompatible with minimal images and must use standard base images. If the run profile does not include -K8S:ImageProfile=standard, big data Jobs using this profile will fail to start.
Save your changes.

Deleting a run profile

Remove a run profile from the system and reassign any tasks that use it to an alternative profile or to run without a profile.

Before you begin

You must have the Run Profiles - Manage (TMC_RUN_PROFILE_MANAGEMENT) user permission to delete run profiles.

Procedure

Open the Processing page.
Click the Run profiles tab.
Right-click the name of the run profile you want to delete then click Delete.
If this run profile is used by existing tasks, reconfigure these tasks by selecting one of the following options:
- Do not use a run profile for these tasks.
- Use an alternative run profile for these tasks and select a run profile from the list.
If you are deleting the last run profile of the engine, the runs will be rescheduled without a run profile, using the default values.
Click Delete to confirm the deletion.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here