Setting up a Dynamic Engine and a Dynamic Engine environment in Kubernetes

Provide one of your own Kubernetes clusters to deploy a Dynamic Engine and its environment or environments on it.

To complete the deployment, you need to:

Create the logical entities for the engine and environments.
Assign environments to the engine.
Generate manifest files for the engine, environments, and their assignments.
Apply those files to your cluster to set up the corresponding physical entities.

The diagram of a Dynamic Engine workflow

Adding a Dynamic Engine

Add a Dynamic Engine logical entity to help deploy a Dynamic Engine in your Kubernetes cluster.

Before you begin

You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role in Talend Management Console.

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Click Add Dynamic Engine.
Enter a name and a description.
The name must contain only alphanumeric, -, . and _ characters. Spaces and other special characters are not allowed.
Click Next.
Optional: A list of available Dynamic Engine environments is displayed. Assign one or more environments from this list or do this later from the Dynamic Engines tab of the Processing page.
Click Create to confirm the Dynamic Engine creation.
The dialog for Dynamic Engine deployment displays.
Select a version from the Version list.
By default, the latest version is selected.

A zip containing two manifest files is generated to apply on your Kubernetes cluster to deploy the Dynamic Engine. You do not need to download this zip yet.

Instead, proceed to create and assign Dynamic Engine environments first. This allows you to download all the required manifest files together.
Close the dialog box.

Results

The status of your Dynamic Engine is set to New.

Adding a Dynamic Engine environment

Create a new environment dedicated to a Dynamic Engine.

This environment can be used with Dynamic Engines only and can be assigned to a single Dynamic Engine at a time.

Before you begin

You must have the Environment Administrator role in Talend Management Console.

Procedure

Go to the Environments page.
Open the Dynamic Engine Environments tab.
Click Add environment.
Enter the name of the environment.
The name must contain only alphanumeric, -, # and _ characters. Leading or trailing spaces and other special characters are not allowed.
Enter the name of the first workspace in the new environment.
The allowed characters are the same as the environment name.
Optional: If you want another user to be the workspace owner, select their username from the drop-down list.
Enter a description.
Click Add to save your changes.

Results

The status of your Dynamic Engine environment is set to Not assigned.

The environment is created with the workspace you specified.

To share the workspace, click the environment name then click the Share workspace icon next to the workspace name from the Workspaces tab.

To add other workspaces to this new environment, click the environment name then Add workspace from the Workspaces tab.

What to do next

You are ready to assign your environment to a Dynamic Engine.

Assigning an environment to a Dynamic Engine and deploying them in Kubernetes

Assign one or more environments to a Dynamic Engine. The next task runs from the environments will occur in the Dynamic Engine.

Assigning an environment to a Dynamic Engine overrides the existing settings of engine and run profile for that environment.

You can assign one or more environments to a Dynamic Engine from the Dynamic Engines tab directly from the list or from the drawer. You can also do it when creating a Dynamic Engine.

Before you begin

You must have the Environment Administrator role in Talend Management Console.
You must have the Infrastructure Administrator role (TMC_CLUSTER_MANAGEMENT) to assign Dynamic Engine environments to a Dynamic Engine.

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Hover over the name of the Dynamic Engine to which you need to assign an environment.
Click the button to open the page for environment assignment.
You can also click the Dynamic Engine name to open the details panel, then click the Manage button.
In the dialog, select one or more environments from the list.
Click Assign.
Click Download the manifest to download the manifest files in a zip.
Unzip the file previously downloaded.
Information noteImportant: Do not change the content of this manifest file. Only the default configuration pre-defined in this file is supported.
In your Kubernetes terminal, run the following commands in the order presented here to set up the Dynamic Engine instance:
1. ```
kubectl apply -f custom-resource-definitions.yaml
```
  This manifest file creates two custom Kubernetes resource definitions in your cluster: qlikruntimeinfras.qlik.com for Dynamic Engine resources and qlikruntimes.qlik.com for Dynamic Engine environment resources.
2. ```
kubectl apply -f resources.yaml
```
  This manifest file creates the Dynamic Engine resources to register your cluster, deploy your Dynamic Engine, and assign the engine to its logical entity in Talend Management Console. These resources belong to a kind of QlikRuntimeInfra.
3. At the end of the second execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=Available=true -n qlik-dynamic-engine deployment/engine-operator-dp --timeout 60s
```
  After waiting for 60 seconds, if the following response appears, it confirms that the engine has been successfully deployed:
```
deployment.apps/engine-operator-dp condition met
```
  In the command above, qlik-dynamic-engine is the namespace that groups the infrastructure services, and deployment/engine-operator-dp is the deployment resource being monitored.
Once done successfully, the status of the Dynamic Engine logic entity in Talend Management Console becomes Active.

If you need to know what has been installed in your Kubernetes cluster, go to your Kubernetes machine and run the commands outlined in this official Kubernetes documentation.
Execute the following command to set up the environments. Only one environment has been created and assigned in this example.
```
kubectl apply -f resources-<environment_id>.yaml
```
Information noteTip:
If the engine is associated with multiple environments, you need this ID to distinguish the correct file.
Two methods are available to find the environment ID:
- In Talend Management Console, open the detail page of the environment to be deleted. The ID is visible in the URL. For example, in this URL
  https://app.eu.cloud.talend.com/tmc/environments/dynamic-engine-environments/66f2c0bb3e063d3518122259/overview
  The 66f2c0bb3e063d3518122259 segment is the environment ID.
- Use the command:
  kubectl get qlikruntime.qlik.com
  This returns the resource name or namespace name with the ID in it. For example, qlik-processing-env-66f2c0bb3e063d3518122259.
At the end of the execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=RUNNING=true --all qlikruntime --timeout 300s
```
During 300 seconds, this command searches for all the QlikRuntime kind of resources. If the response similar to the following one appears, it confirms that the engine environment has been successfully deployed:
```
qlikruntime.qlik.com/qlik-processing-env-66f2c0bb3e063d3518122259 condition met
```
In this response, the name of the resource being monitored is qlik-processing-env-664b0bbd6c6f5522732e055d. The name is also used as the namespace name, grouping all the services associated with the Dynamic Engine environment resource.

Optional: If you did not obtain the confirmation above, the deployment has issues. Run the following command to inspect the "serviceStatus" object of the environment resource:

kubectl get qlikruntime qlik-processing-env-664b0bbd6c6f5522732e055d -o json | jq -r '.status'

In this command, qlikruntime represents the kind of custom resources dedicated to Dynamic Engine environments, and as is shown in the previous example, qlik-processing-env-664b0bbd6c6f5522732e055d is the resource and the namespace name of the environment.

The following message shows an example of the error on the "di-job-deployer" service in the "serviceStatus" object. Its "jobErrorMessage" field gives more details about the error observed during the deployment, indicating that the expected helm chart has not been found.

Note that some lines are skipped in the following example.

{
  "conditions": [
    {
      "lastTransitionTime": "2024-05-30T13:05:25Z",
      "message": "Service is not deployed",
      "reason": "Deployment_not_found",
      "status": "True",
      "type": "NOT_DEPLOYED"
    }
  ],
  "serviceStatus": {
    ...
    "di-job-deployer": {
      "conditions": [
        ...
        {
          "lastTransitionTime": "2024-05-30T13:06:39Z",
          "message": "Service is in error",
          "reason": "Pod_failed",
          "status": "False",
          "type": "IN_ERROR"
        }
      ],
      "creationDate": "2024-05-30T13:05:25Z",
      "jobErrorMessage": "Error: release: not found\nUpgrading...\nupgrade 
      result: Release \"di-job-deployer-92d6e127-e070-4264-a9e8-00adbbbfbc03\" does not exist. 
      Installing it now.\nError: failed to download 
      ...
      \nError: failed to download 
      \"https://artifactory.datapwn.com/artifactory/tlnd-helm-ce-dev/dpe-di-job-deployer-2.0.0.tgz\
      "\nexit_code: 1\n",
      "jobStatus": "FAILED",
      "lastUpdateDate": "2024-05-30T13:01:52Z"
    },
      ...
    }
  }
}

In this situation, contact the Support team to investigate the Dynamic Engine environment manifest file. When informed, unassign the environment and assign it again to generate a new manifest file.

Results

Once done successfully, the Dynamic Engine environment services are installed in your cluster and connect to the Dynamic Engine.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If the manifest implementation fails or the Dynamic Engine services are disassociated from the Dynamic Engine, the status become Not ready.

What to do next

Since the Dynamic Engine has been successfully deployed in your Kubernetes, connected to Talend Management Console, and associated with a Dynamic Engine environment, it is ready to run Job tasks or plans as you usually do on the standard engines. For details about how to add the task to the Dynamic Engine environment for a run, see Adding tasks to a Dynamic Engine environment.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here