Implementing Dynamic Engine on a minikube for hands-on testing

Beta

Install and configure Dynamic Engines on a minikube cluster for a quick hands-on before implementing them in your organization.

Prerequisites:

You must have Docker and minikube installed.
Ensure that the Docker daemon is running.
The minikube tool has been installed and configured to use a Kubernetes version supported by Dynamic Engines. For further information about the supported Kubernetes versions, see Kubernetes specifications.
For further information, see minikube installation.

Add a Dynamic Engine on your minikube cluster

Beta

Procedure

Run the following command in your terminal to start a minikube instance:
```
minikube start -p QlikDE --cpus=4 --memory=8000
```
This command will start an instance named QlikDE and allocate 4 CPUs and 8 GB of memory. You can use any name for the instance.
Go to the Processing page in Talend Management Console, select the Dynamic Engines tab, and add a new Dynamic Engine.
This is only the logical entity of your Dynamic Engine.
Download the manifest file to the minikube machine to set up the physical entity of the engine.
For detailed instructions, see this procedure.
Unzip the downloaded manifest file and run the following commands to apply the manifests:
- ```
kubectl apply -f {download_dir}/qlik/custom-resource-definitions.yaml
```
  This manifest file creates two custom Kubernetes resource definitions in your cluster: dynamicengines.apps.qlik.com for Dynamic Engine resources and dynamicengineenvs.apps.qlik.com for Dynamic Engine environment resources.
- ```
kubectl apply -f {download_dir}/qlik/resources.yaml
```
  This manifest file creates the Dynamic Engine resources to register your cluster, deploy your Dynamic Engine, and assign the engine to its logical entity in Talend Management Console. These resources belong to a kind of DynamicEngine.
Verify if the manifest files were successfully applied by running this command:
```
kubectl get crd
```
You should see entries like this, indicating that custom resources have been successfully defined.
```
NAME                               CREATED AT
dynamicengines.apps.qlik.com         2024-08-15T14:51:37Z
dynamicengineenvs.apps.qlik.com              2024-08-15T14:51:37Z
```

Results

Deploying and starting the engine can take some time. When complete, its logic entity in Talend Management Console shows as active.

To monitor the process, run the following command:

kubectl wait --for=condition=Ready pod -l 'app.qlik.com/owned-by' --timeout 10m -A

This command verifies the readiness of all pods with the label app.qlik.com/owned-by across all namespaces.

If the deployment is successful, a response similar to the following is displayed:

pod/dynamic-engine-operator-799bb96687-rfm2g condition met
pod/dpe-data-service-route-deployer-74b7f566c-lrl99 condition met
pod/dpe-di-job-deployer-5cff46b6-x5t85 condition met
pod/dpe-di-job-restore-orphans-5f57f5f86f-d9vpb condition met
pod/dpe-engine-config-manager-956d5d7b9-7krcn condition met
pod/dpe-job-controller-66fd596588-7x5w5 condition met
pod/dpe-log-collector-7dd9499c8-dnvrn condition met

This output confirms that all the pods required for Dynamic Engine and its environment are ready.

Add a Dynamic Engine environment for the Dynamic Engine

Beta

Procedure

In Talend Management Console, create a Dynamic Engine environment logical entity. For further information, see Adding a Dynamic Engine environment.
Go to the Processing page and open the Dynamic Engines tab.
Select the Dynamic Engine you created previously in Talend Management Console.
Click the button to open the environment assignment page.
Select the environment you created and click Assign.
Download the manifest file of this environment.
Apply the manifest file in your minikube machine by running:
```
kubectl apply -f {download_dir}/qlik/resources-<environment_id>.yaml
```
The entire process takes 3 to 4 minutes depending on the capacity of your minikube machine. You can use the following commands to verify the progress:
- kubectl wait --for=condition=Ready pod -A -l 'app.qlik.com/part-of' --timeout=10m
  This command verifies if all environment related pods are all set up and running. Upon completion, a message like this is returned:
  pod/dynamic-engine-operator-68478645bb-wn7x9 condition met pod/dpe-di-job-deployer-86dc4ddb7f-7v9lz condition met pod/dpe-di-job-restore-orphans-6d4f8968d4-g75dk condition met pod/dpe-engine-config-manager-6bc9fb44-4z6nh condition met pod/dpe-job-controller-7c74467648-p7k8c condition met pod/dpe-log-collector-7fc4fb664d-scdf9 condition met
  Pods running in other namespaces are also listed in the message. For example, dynamic-engine-operator-68478645bb-wn7x9.
- kubectl get namespaces
  This command lists all the namespaces, allowing you to identify the namespace of the Dynamic Engine environment, which groups all associated services, such as qlik-processing-env-66f2c0bb3e063d3518122259.
- kubectl get pods --namespace qlik-processing-env-66f2c0bb3e063d3518122259
  This command lists all the service of the environment being deployed. When all their statuses show Running, the installation of your Dynamic Engine environment is completed. In Talend Management Console, the status of this environment changes to Ready, indicating that it is ready to run tasks or plans.

Results

Follow this procedure to run tasks in this Dynamic Engine environment.

What to do next

If you want to delete the minikube instance, use the following command:

minikube delete --profile QlikDE

This will remove the minikube instance and all its associated files.

For more information about Dynamic Engine logs: see Deactivating log transfer.
For known limitations of Dynamic Engines, see Known limitations.
For a complete guide about Dynamic Engines, see Dynamic Engine Configuration Guide. With this guide, you create and deploy Dynamic Engines and their environments for production.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here