Deploying Dynamic Engine with existing PersistentVolumeClaims

Configure the Dynamic Engine Helm charts to reference existing PersistentVolumeClaims by name instead of using dynamic StorageClass provisioning.

Before you begin

The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd Helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.
The three PersistentVolumeClaim resources (job-data, archive, and custom-resources) are created and in Bound state in the environment namespace. If you are using docker-registry, the docker-registry claim must also be in Bound state in the engine namespace.

About this task

Use this procedure when storage is already provisioned in your cluster, for example after setting up EKS with Amazon S3 (see Setting up Amazon EKS with S3) or another pre-provisioned storage system. In this model, the Dynamic Engine Helm charts bind to named PersistentVolumeClaims at deployment time rather than requesting storage from a StorageClass. This is the static provisioning model. For the dynamic provisioning model, where Kubernetes creates volumes automatically from a StorageClass, see Provisioning a storage class dedicated to Dynamic Engine environment services.

Both the Dynamic Engine chart (for the embedded registry volume) and the Dynamic Engine Environment chart (for the three processing volumes) must be configured separately.

Procedure

Set the environment variables for your deployment:

export DYNAMIC_ENGINE_VERSION=<your-dynamic-engine-version>
export DYNAMIC_ENGINE_ID=<your-dynamic-engine-id>
export DYNAMIC_ENGINE_ENVIRONMENT_ID=<your-dynamic-engine-environment-id>

Replace the placeholder values with your actual Dynamic Engine version, engine ID, and environment ID.

Verify that the required PersistentVolumes and PersistentVolumeClaims exist and are in Bound state:

kubectl get pv

Confirm that the output lists the expected volumes with Bound status and the correct claim namespaces. For example:

NAME               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
docker-registry    1200Gi     RWX            Retain           Bound    qlik-dynamic-engine/docker-registry
archive            1200Gi     RWX            Retain           Bound    qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID/archive
custom-resources   1200Gi     RWX            Retain           Bound    qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID/custom-resources
job-data           1200Gi     RWX            Retain           Bound    qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID/job-data

Note: The 1200Gi capacity value is required by the Kubernetes API but is not enforced. For S3-backed volumes, actual storage capacity is determined by the S3 bucket, not by this field.

Configure and deploy the Dynamic Engine chart

Create a custom values file for the Dynamic Engine chart, specifying the docker-registry claim name:

cat <<EOF > $DYNAMIC_ENGINE_ID-custom-set-static-pvc-values.yaml
configuration:
  dockerRegistry:
    persistence:
      existingClaim: docker-registry
EOF

Deploy the Dynamic Engine using the base values file and the custom values file:

helm upgrade --install dynamic-engine-$DYNAMIC_ENGINE_ID \
  -n qlik-dynamic-engine \
  -f $DYNAMIC_ENGINE_ID-values.yaml \
  -f $DYNAMIC_ENGINE_ID-custom-set-static-pvc-values.yaml \
  oci://ghcr.io/talend/helm/dynamic-engine \
  --version $DYNAMIC_ENGINE_VERSION

Configure and deploy the Dynamic Engine Environment chart

Create a custom values file for the Dynamic Engine Environment chart, specifying the three processing volume claim names:
```
cat <<EOF > $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-set-static-pvc-values.yaml
configuration:
  persistence:
    persistentVolumeClaims:
      job_data:
        persistentVolumeClaimName: job-data
      archive:
        persistentVolumeClaimName: archive
      custom_resources:
        persistentVolumeClaimName: custom-resources
EOF
```
Information noteNote: The Helm chart API requires underscores for the persistentVolumeClaims keys (job_data, custom_resources). The corresponding PersistentVolumeClaim names use hyphens (job-data, custom-resources). This asymmetry is intentional.

Deploy the Dynamic Engine Environment using the base values file and the custom values file:

helm upgrade --install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
  -n qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
  -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml \
  -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-set-static-pvc-values.yaml \
  oci://ghcr.io/talend/helm/dynamic-engine-environment \
  --version $DYNAMIC_ENGINE_VERSION

Results

Dynamic Engine and its environment are now running with the existing PersistentVolumeClaims. Confirm that the docker-registry pod in the qlik-dynamic-engine namespace and the dpe-di-job-deployer pod in the qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID namespace are running.

What to do next

Run the following commands to confirm that the pods are running in both namespaces:

kubectl get pods -n qlik-dynamic-engine
kubectl get pods -n qlik-processing-env-$DYNAMIC_ENGINE_ENVIRONMENT_ID

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here