Setting up GKE with Filestore storage for Dynamic Engine

After your GKE cluster is ready, configure Filestore-backed shared storage and deploy it with a Dynamic Engine environment.

Dynamic Engine requires this shared storage only when configuration.persistence.enabled=true is configured for your engine. Otherwise, the storage is optional and becomes necessary only when your Talend Management Console tasks need persistent storage.

Before you begin

A GKE cluster configured for Dynamic Engine deployment. For platform-specific prerequisites, cluster configuration, and cluster access steps, see Configuring Google Kubernetes Engine (GKE) for Dynamic Engine deployment.
The dynamic-engine-crd custom resource definitions must have been installed using the oci://ghcr.io/talend/helm/dynamic-engine-crd Helm chart. If not, run the following commands for the installation:
1. Find the chart version to be used:
  - Run the following Helm command:
    helm show chart oci://ghcr.io/talend/helm/dynamic-engine-crd --version <engine_version>
  - See the version directly from Talend Management Console or check the Dynamic Engine changelog for the chart version included in your Dynamic Engine version.
  - Use an API call to the Dynamic Engine version endpoint.
2. Run the following command to install the Helm chart of a given version:
```
helm install dynamic-engine-crd oci://ghcr.io/talend/helm/dynamic-engine-crd --version <helm_chart_version>
```
  Replace <helm_chart_version> with the chart version supported by your Dynamic Engine version.
  Without specifying the version, you install the latest available dynamic-engine-crd chart version.

About this task

Important: For GKE Autopilot deployments, it is recommended to first apply the managed-mode configuration set from Managed Kubernetes operating modes before running these steps.

This procedure describes how to configure Filestore-backed shared storage and deploy a customized Dynamic Engine environment in a GKE cluster.

Procedure

Configure a ReadWriteMany GKE storage class for Dynamic Engine.
Create a ReadWriteMany storage class dedicated to Dynamic Engine, and in later steps, use its name as defaultStorageClassName. For example:
```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dyn-engine
provisioner: filestore.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  tier: <filestore-tier>
  network: <name-of-configured-network>
```
This example creates a storage class called dyn-engine, for which the provisioner filestore.csi.storage.gke.io is configured to create persistence volumes of a given type in a given network, respectively specified in the tier and network fields.
- For supported Filestore values in tier, see GKE tier.
- Use your configured GKE network name. If no custom network is configured, use default. If you remove the network field, default is used.
Information noteTip: A dedicated storage class for Dynamic Engine is recommended, but it is still possible to use the default storage classes provided by GKE, depending on your organization's policy. For a list of these default storage classes, see Automatically installed GKE storage class.
Deploy the Dynamic Engine instance and its environment.
Information noteNote: The steps below demonstrate the unified Dynamic Engine environment deployment pattern. For detailed instructions on configuring custom storage classes across all platforms (AWS, GKE, AKS, and on-premises Kubernetes), see Provisioning a storage class dedicated to Dynamic Engine environment services.
1. Run this command to deploy the engine instance:
```
helm upgrade --install dynamic-engine \
  -f $DYNAMIC_ENGINE_ID-helm-values/$DYNAMIC_ENGINE_ID-values.yaml \
  oci://ghcr.io/talend/helm/dynamic-engine
```
  This is a default deployment. This command upgrades or installs the Dynamic Engine instance depending on whether it has been deployed. Replace $DYNAMIC_ENGINE_ID with the Dynamic Engine ID, for example, c-m-sjufu4qy.
2. Create a custom Helm values file for the environment.
  Example
```
cat <<EOF > $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-gke-values.yaml
---
configuration:
  persistence:
    defaultStorageClassName: dyn-engine
EOF
```
  This example sets dyn-engine, the storage class you previously created, as the name of defaultStorageClassName. Change it according to the Dynamic Engine-specific storage class you are using.
3. Deploy the Helm charts for the Dynamic Engine environment.
```
helm upgrade --install dynamic-engine-environment-$DYNAMIC_ENGINE_ENVIRONMENT_ID \
  -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-values.yaml  \
  -f $DYNAMIC_ENGINE_ENVIRONMENT_ID-custom-gke-values.yaml \
  oci://ghcr.io/talend/helm/dynamic-engine-environment \
  --version $DYNAMIC_ENGINE_VERSION
```
  This command upgrades or installs the Dynamic Engine environment depending on whether it has been deployed.

Results

Once complete, the Dynamic Engine environment services are installed in your GKE cluster and are ready to run tasks or plans.

In Talend Management Console, the status of this environment becomes Ready, confirming that it is ready to run tasks or plans.

If deployment fails, or if Dynamic Engine services are disassociated, the status becomes Not ready.

What to do next

After successful deployment, you can add tasks to the Dynamic Engine environment as you would for standard engines. For details, see Adding a Job task in a Dynamic Engine environment.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here