Kubectl deployment

Apply the manifest files for the deployment of the Dynamic Engine ecosystem.

This is the classic way of deploying Dynamic Engine services. Unlike Helm, Kubectl does not allow custom deployment configuration.

Tip: About Helm and Kubectl:

They are two options for the deployment of your Dynamic Engine ecosystem.

If you choose Helm, skip this section and see (Recommended) Helm deployment.
If you want to change the selected deployment type for your Dynamic Engine and its environment, see Changing the deployment type of a Dynamic Engine.

Upgrading from Kubernetes manifests to Helm charts is not supported for already deployed Dynamic Engine services.

Before you begin

You must have assigned at least one Dynamic Engine environment to your engine, selected the Kubectl deployment type, and downloaded the zip file. For instructions about this procedure, see Assigning an environment to a Dynamic Engine.

Procedure

Unzip the file previously downloaded.
In your Kubernetes terminal, run the following commands in the order presented here to set up the Dynamic Engine instance:
1. ```
kubectl apply -f custom-resource-definitions.yaml
```
  This manifest file creates two custom Kubernetes resource definitions in your cluster: dynamicengines.apps.qlik.com for Dynamic Engine resources and dynamicengineenvs.apps.qlik.com for Dynamic Engine environment resources.
2. ```
kubectl apply -f resources.yaml
```
  This manifest file creates the Dynamic Engine resources to register your cluster, deploy your Dynamic Engine, and assign the engine to its logical entity in Talend Management Console. These resources belong to a kind of DynamicEngine.
3. At the end of the second execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=Ready pod -l 'app.qlik.com/owned-by' --timeout 10m -A 
```
  This command verifies the readiness of all pods with the label app.qlik.com/owned-by across all namespaces.
  
  If the deployment is successful, a response similar to the following is displayed:
```
pod/dynamic-engine-operator-799bb96687-rfm2g condition met
                           pod/dpe-data-service-route-deployer-74b7f566c-lrl99 condition met
                           pod/dpe-di-job-deployer-5cff46b6-x5t85 condition met
                           pod/dpe-di-job-restore-orphans-5f57f5f86f-d9vpb condition met
                           pod/dpe-engine-config-manager-956d5d7b9-7krcn condition met
                           pod/dpe-job-controller-66fd596588-7x5w5 condition met
                           pod/dpe-log-collector-7dd9499c8-dnvrn condition met      
```
  This output confirms that all the pods required for Dynamic Engine and its environment are ready.
Once done successfully, the status of the Dynamic Engine logic entity in Talend Management Console becomes Active.

If you need to know what has been installed in your Kubernetes cluster, go to your Kubernetes machine and run the commands outlined in this official Kubernetes documentation.
Execute the following command to set up the environments. Only one environment has been created and assigned in this example.
```
kubectl apply -f resources-<environment_id>.yaml
```
Information noteTip:
If the engine is associated with multiple environments, you need this ID to distinguish the correct file.
Two methods are available to find the environment ID:
- In Talend Management Console, open the detail page of the environment to be deleted. The ID is visible in the URL. For example, in this URL
  https://app.eu.cloud.talend.com/tmc/environments/dynamic-engine-environments/66f2c0bb3e063d3518122259/overview
  The 66f2c0bb3e063d3518122259 segment is the environment ID.
- Use the command:
  kubectl get dynamicengineenvs.qlik.com
  This returns the resource name or namespace name with the ID in it. For example, qlik-processing-env-66f2c0bb3e063d3518122259.
At the end of the execution, run the following command to verify the deployment:
```
kubectl wait --for=condition=RUNNING=true --all dynamicengineenvs --timeout 300s
```
During 300 seconds, this command searches for all the DynamicEngineEnv kind of resources. If the response similar to the following one appears, it confirms that the engine environment has been successfully deployed:
```
dynamicengineenvs.qlik.com/qlik-processing-env-66f2c0bb3e063d3518122259 condition met
```
In this response, the name of the resource being monitored is qlik-processing-env-664b0bbd6c6f5522732e055d. The name is also used as the namespace name, grouping all the services associated with the Dynamic Engine environment resource.

Optional: If you did not obtain the confirmation above, the deployment has issues. Run the following command to inspect the "serviceStatus" object of the environment resource:

kubectl get dynamicengineenvs qlik-processing-env-664b0bbd6c6f5522732e055d -o json | jq -r '.status'

In this command, dynamicengineenvs represents the kind of custom resources dedicated to Dynamic Engine environments, and as is shown in the previous example, qlik-processing-env-664b0bbd6c6f5522732e055d is the resource and the namespace name of the environment.

The following message shows an example of the error on the "di-job-deployer" service in the "serviceStatus" object. Its "jobErrorMessage" field gives more details about the error observed during the deployment, indicating that the expected helm chart has not been found.

Note that some lines are skipped in the following example.

{
                           "conditions": [
                           {
                           "lastTransitionTime": "2024-05-30T13:05:25Z",
                           "message": "Service is not deployed",
                           "reason": "Deployment_not_found",
                           "status": "True",
                           "type": "NOT_DEPLOYED"
                           }
                           ],
                           "serviceStatus": {
                           ...
                           "di-job-deployer": {
                           "conditions": [
                           ...
                           {
                           "lastTransitionTime": "2024-05-30T13:06:39Z",
                           "message": "Service is in error",
                           "reason": "Pod_failed",
                           "status": "False",
                           "type": "IN_ERROR"
                           }
                           ],
                           "creationDate": "2024-05-30T13:05:25Z",
                           "jobErrorMessage": "Error: release: not found\nUpgrading...\nupgrade 
                           result: Release \"di-job-deployer-92d6e127-e070-4264-a9e8-00adbbbfbc03\" does not exist. 
                           Installing it now.\nError: failed to download 
                           ...
                           \nError: failed to download 
                           \"https://artifactory.datapwn.com/artifactory/tlnd-helm-ce-dev/dpe-di-job-deployer-2.0.0.tgz\
                           "\nexit_code: 1\n",
                           "jobStatus": "FAILED",
                           "lastUpdateDate": "2024-05-30T13:01:52Z"
                           },
                           ...
                           }
                           }
                           }

In this situation, contact the Support team to investigate the Dynamic Engine environment manifest file. When informed, unassign the environment and assign it again to generate a new manifest file.

Results

Once done successfully, the Dynamic Engine environment services are installed in your cluster and connect to the Dynamic Engine.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If the manifest implementation fails or the Dynamic Engine services are disassociated from the Dynamic Engine, the status become Not ready.

What to do next

Since the Dynamic Engine has been successfully deployed in your Kubernetes, connected to Talend Management Console, and associated with a Dynamic Engine environment, it is ready to run Job tasks or plans as you usually do on the standard engines. For details about how to add the task to the Dynamic Engine environment for a run, see Adding tasks to a Dynamic Engine environment.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here