Setting up a Dynamic Engine and a Dynamic Engine environment in Kubernetes

Provide one of your own Kubernetes clusters to deploy a Dynamic Engine and its environment on it.

Adding a Dynamic Engine

Add a Dynamic Engine logical entity to help deploy a Dynamic Engine in your Kubernetes cluster.

Before you begin

You must have the Infrastructure Administrator (containing the TMC_CLUSTER_MANAGEMENT permission) role in Talend Management Console.

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Click Add Dynamic Engine.
Enter a name and a description.
The name must contain only alphanumeric, -, . and _ characters. Spaces and other special characters are not allowed.
Click Next.
Optional: Assign one or more environments.
You can also assign environments later from the Dynamic Engines tab of the Processing page.
Click Create to confirm the Dynamic Engine creation.
The dialog for Dynamic Engine deployment displays.
A zip containing two manifest files is generated to apply on your Kubernetes cluster to deploy the Dynamic Engine.
Download the manifest zip.
You can always find the link to download this file by hovering the cursor over the Dynamic Engine to be deployed on the Dynamic Engines list, until you have deployed the engine.
Close the dialog box.

Results

A zip file is generated. You need to download it and then sequentially apply the two manifest files in it on your Kubernetes cluster to register the cluster and deploy your Dynamic Engine.

Important: Do not change the content of this manifest file. Only the default configuration pre-defined in this file is supported.

The status of your Dynamic Engine is set to New.

Setting up your Dynamic Engine in Kubernetes using the manifest files

Apply the generated Dynamic Engine manifest files in the Kubernetes cluster to deploy the Dynamic Engine.

Procedure

Unzip the file previously downloaded to obtain two manifest files: custom-resource-definitions.yaml and resources.yaml.
In your Kubernetes terminal, run the following commands to apply the manifest files in the order presented here:
1. ```
kubectl apply -f custom-resource-definitions.yaml
```
  This manifest file creates two custom Kubernetes resource definitions in your cluster: QlikRuntimeInfra for Dynamic Engine resources and QlikRuntime for Dynamic Engine environment resources.
2. ```
kubectl apply -f resources.yaml
```
  This manifest file creates the Dynamic Engine resources to register your cluster, deploy your Dynamic Engine, and assign the engine to its logical entity in Talend Management Console.
At the end of the second execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=Available=true -n qlik-dynamic-engine deployment/engine-operator-dp --timeout 60s
```
After waiting for 60 seconds, if the following response appears, it confirms that the engine has been successfully deployed:
```
deployment.apps/engine-operator-dp condition met
```
In the command above, qlik-dynamic-engine is the namespace that groups the infrastructure services, and deployment/engine-operator-dp is the deployment resource being monitored.

Results

Once done successfully, the status of the Dynamic Engine logic entity in Talend Management Console becomes Active.

If you need to know what has been installed in your Kubernetes cluster, go to your Kubernetes machine and run the commands outlined in this official Kubernetes documentation.

Adding a Dynamic Engine environment

Create a new environment dedicated to a Dynamic Engine.

This environment can be used with Dynamic Engines only and can be assigned to a single Dynamic Engine at a time.

Before you begin

You must have the Environment Administrator role in Talend Management Console.

Procedure

Go to the Environments page.
Open the Dynamic Engine Environments tab.
Click Add environment.
Enter the name of the environment.
The name must contain only alphanumeric, -, # and _ characters. Leading or trailing spaces and other special characters are not allowed.
Enter the name of the first workspace in the new environment.
The allowed characters are the same as the environment name.
Optional: If you want another user to be the workspace owner, select their username from the drop-down list.
Enter a description.
Click Add to save your changes.

Results

The status of your Dynamic Engine environment is set to Not assigned.

The environment is created with the workspace you specified.

To share the workspace, click the environment name then click the Share workspace icon next to the workspace name from the Workspaces tab.

To add other workspaces to this new environment, click the environment name then Add workspace from the Workspaces tab.

What to do next

You are ready to assign your environment to a Dynamic Engine.

Assigning an environment to a Dynamic Engine

Assign one or more environments to a Dynamic Engine. The next task runs from the environments will occur in the Dynamic Engine.

Assigning an environment to a Dynamic Engine overrides the existing settings of engine and run profile for that environment.

You can assign one or more environments to a Dynamic Engine from the Dynamic Engines tab directly from the list or from the drawer. You can also do it when creating a Dynamic Engine.

Before you begin

You must have the Environment Administrator role in Talend Management Console.
You must have the Infrastructure Administrator role (TMC_CLUSTER_MANAGEMENT) to assign Dynamic Engine environments to a Dynamic Engine.

Procedure

Go to the Processing page.
Open the Dynamic Engines tab.
Hover over the name of the Dynamic Engine to which you need to assign an environment.
Click the button to open the page for environment assignment.
You can also click the Dynamic Engine name to open the details panel, then click the Manage button.
In the dialog, select one or more environments from the drop-down list.
Click Assign.
Click Download the manifest to download this manifest file.
In the terminal of your Kubernetes cluster, execute the following command to apply this manifest file.
```
kubectl apply -f <your_environment_manifest_file>.yaml
```
At the end of the execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=RUNNING=true --all qlikruntime --timeout 300s
```
During 300 seconds, this command searches for all the resources of the QlikRuntime type. If the response similar to the following one appears, it confirms that the engine environment has been successfully deployed:
```
qlikruntime.qlik.com/qlik-processing-env-66f2c0bb3e063d3518122259 condition met
```
In this response, the name of the resource being monitored is qlik-processing-env-664b0bbd6c6f5522732e055d. The name is also used as the namespace name, grouping all the services associated with the Dynamic Engine environment resource.

Optional: If you did not obtain the confirmation above, the deployment has issues. Run the following command to inspect the "serviceStatus" object of the environment resource:

kubectl get qlikruntime qlik-processing-env-664b0bbd6c6f5522732e055d -o json | jq -r '.status'

In this command, qlikruntime represents the custom resource type dedicated to Dynamic Engine environments, and as is shown in the previous example, qlik-processing-env-664b0bbd6c6f5522732e055d is the resource and the namespace name of the environment.

The following message shows an example of the error on the "di-job-deployer" service in the "serviceStatus" object. Its "jobErrorMessage" field gives more details about the error observed during the deployment, indicating that the expected helm chart has not been found.

Note that some lines are skipped in the following example.

{
  "conditions": [
    {
      "lastTransitionTime": "2024-05-30T13:05:25Z",
      "message": "Service is not deployed",
      "reason": "Deployment_not_found",
      "status": "True",
      "type": "NOT_DEPLOYED"
    }
  ],
  "serviceStatus": {
    ...
    "di-job-deployer": {
      "conditions": [
        ...
        {
          "lastTransitionTime": "2024-05-30T13:06:39Z",
          "message": "Service is in error",
          "reason": "Pod_failed",
          "status": "False",
          "type": "IN_ERROR"
        }
      ],
      "creationDate": "2024-05-30T13:05:25Z",
      "jobErrorMessage": "Error: release: not found\nUpgrading...\nupgrade 
      result: Release \"di-job-deployer-92d6e127-e070-4264-a9e8-00adbbbfbc03\" does not exist. 
      Installing it now.\nError: failed to download 
      ...
      \nError: failed to download 
      \"https://artifactory.datapwn.com/artifactory/tlnd-helm-ce-dev/dpe-di-job-deployer-2.0.0.tgz\
      "\nexit_code: 1\n",
      "jobStatus": "FAILED",
      "lastUpdateDate": "2024-05-30T13:01:52Z"
    },
      ...
    }
  }
}

In this situation, contact the Support team to investigate the Dynamic Engine environment manifest file. When informed, unassign the environment and assign it again to generate a new manifest file.

Results

Once done successfully, the Dynamic Engine environment services are installed in your cluster and connect to the Dynamic Engine.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If the manifest implementation fails or the Dynamic Engine services are disassociated from the Dynamic Engine, the status become Not ready.

What to do next

Since the Dynamic Engine has been successfully deployed in your Kubernetes, connected to Talend Management Console, and associated with a Dynamic Engine environment, it is ready to run Job tasks or plans as you usually do on the standard engines. For details about how to add the task to the Dynamic Engine environment for a run, see Adding tasks to a Dynamic Engine environment.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here